Log Correlation Engine 6.0.1 Release Notes - 2019-06-25

Note: Before upgrading 5.0.x to 6.0.x, upgrade and migrate to 5.1.1.

New Features

  • New command-line utilities, all under /opt/lce/tools/:

    • ts-test, for checking how a particular log would be tokenized for the purpose of text search indexing, and whether a particular text search phrase would match it.

    • validate-PRM-regex, for checking whether a regex as specified in a custom .prm definition would match a particular log.

  • New helper scripts, all under /opt/lce/tools/pg-helper-sql/:

    • disk-usage-summary.sql, gives a concise summary of disk usage by table category (tables which store events, tables which maintain filter pointers, rollup counts tables, etc.).

      • Output of this script has been added to diag report, to facilitate troubleshooting.

    • drop-indexes-on-older-silos.sql, allows operator to easily free up disk space by dropping indexes on silos which have not yet been archived/trimmed out of activeDb but are no longer queried. (Indexes can be easily regenerated later if needed.)

    • coverage-by-hhour.sql, displays a concise infographic of rollup counts coverage of the events in a given silo, with half-hourly granularity.

      • Output of this script has been added to diag report, to facilitate troubleshooting.

    • presence-dim-by-hhour.sql, displays a concise infographic of the presence of a particular type (or user, or sensor, or ...) among the events in a given silo, with half-hourly granularity. This can provide a bare-bones reporting capacity even when Tenable.sc connection is interrupted.

Changed Functionality and Performance Enhancements

  • Optimization: queries filtering on single IP address now take advantage of indexes on the src_ip and/or dst_ip columns of events table, obviating use of the filter pointers mechanism which has a higher overhead.

  • Optimization: queries over extensive time ranges now can leverage consistent sampling, trading marginal accuracy for significant speedup.

  • The full-processes.sql and watch-processes.sql scripts now correctly group PostgreSQL backend processes by client, then by work group.

  • The table-sizes.sql script now breaks down each table's disk usage by free space map, transaction visibility map, in-line tuples, indexes on in-line tuples, out-of-line data, and indexes on out-of-line data.

  • The tasl.log report of .tasl and .nbin script execution statistics is now first emitted 10 minutes after lce_tasld startup; and it can also be emitted on command at any time thereafter, upon receipt of SIGRTMIN+10 signal.

  • The X_hhourly tables are now LOGGED, and hence will not lose data in the event of abnormal termination of the main PostgreSQL process.

  • The source-for-psql-shortcuts.sh script now automatically sets the PGTZ environment variable; this ensures that timestamps, in the results of queries invoked via /opt/lce/postgresql/bin/psql, are shown in the correct timezone.

Bug Fixes

  • Windows OS version of LCE client hosts was being shown as Windows 2008 in Web UI, regardless of actual Windows OS version.

  • Data Migration from LCE 5 could fail, under a narrow set of circumstances.

  • Type (or user, or sensor, or ...) literals were not being properly substituted in the responses to showids queries, in some cases.

  • Scanning of new events by lce_tasld daemon did not keep pace with creation of new events by lced daemon.

  • Scanning of new events by lce_tasld daemon could fail to progress to a subsequent silo.

  • Scanning of new events by stats daemon could fail to progress to a subsequent silo.

  • Text queries containing stopwords and/or SQL-reserved punctuation did not return expected results.

  • The optimize-datastore utility would take too long to complete, because it did not skip already-processed tables.

  • Parallel processes forked by the optimize-datastore utility could cause inordinate contention.

  • Needlessly aggressive cancellation of executing SQL queries was being done during silo roll.

Supported Platforms

  • Red Hat Enterprise Linux 5 64-bit / CentOS 5 64-bit
  • Red Hat Enterprise Linux 6 64-bit / CentOS 6 64-bit
  • Red Hat Enterprise Linux 7 64-bit / CentOS 7 64-bit