Log Correlation Engine 6.0.4 Release Notes (05-11-2020)

New Features

  • Direct-from-4X migration, using the same migrateDB-overseer interface used for direct-from-5X migration.
  • HA support, configured with the new utility ha-manager.
  • New utility online-pg-backup, for online backup (also known as hot backup) of LCE's PostgreSQL database.
  • New utility port-controlfiles, to aid moving an LCE instance from one host to another.
  • The alert occasion code, associated with alerts as of 6.0.3, is now displayed in Web UI where alerts are listed.
  • An activeDb_disk_device_I_O_saturated alert will now be created when conditions warrant, to give you better visibility into resource thresholds.
  • New option --vlike for the cfg-utils utility; it neither matches by nor shows old config attribute name, but does show (up to 50 characters of) the value of each matched config attribute.
  • New helper script alerts-by-month.sql: alerts-by-day.sql's longer-term counterpart.

Changed Functionality and Performance Enhancements

  • Increased throughput, compared to previous versions, when the multi-valued configuration attribute syslog_sensors has many values.
  • The optimize-datastore utility now resumes work more promptly, after a pause meant to prevent spikes in resource consumption; as consequence, optimize-datastore now needs less time per silo.
  • With 30 seconds cut from the lced daemon's shutdown time, restarts are quicker and events loss window is smaller.

Bug Fixes

  • When estimating disk size taken up by activeDb, the lced daemon would consider the entire activeDb directory, instead of just its postgresql/ subdirectory; at installations with pre-6X silos still present and stored in the activeDb directory, this would result in an over-estimate.

  • The lced daemon would manifest a severe memory leak when configured to forward syslog over UDP.

  • The lced daemon could fail to restart after having been abnormally terminated (e.g. due to sudden host power-off) while a silo roll was in progress.

  • In the very rare case of PostgreSQL being restarted right after startup of the lced daemon, one of the lced daemon's dbWriter threads could fail to persist events.

  • Generation of diag report could hang while recording sample of input to the LCE Netflow client, if that client is installed on same host.

  • Diagnostics could be launched multiple times from the UI and link to report could be clicked before diag actually finished.

  • When upgrading from pre-6.0.3 installations, values of the event_rules configuration attribute would not be correctly migrated.

  • Prevent inadvertent concurrent execution of the diag utility.

  • Network name of client hosts not populated.

  • Timestamp of alerts as shown in WebUI would be displayed in UTC and not localtime.

  • The cfg-utils utility would not accept values with embedded newlines.

  • Upgrade from 6.0.0/6.0.1/6.0.2 would fail if the postgresql service had not been running at the time rpm -U was invoked.

  • When upgrading from pre-6.0.3 installations, values of syslog_forward_destinations__TCP configuration attribute would not be correctly migrated.

  • WebUI descriptions of several configuration attributes' required format were inaccurate and misleading.

  • Only the first of multiple configured syslog sensors would be used by the lced daemon.

  • The optimize-datastore utility could continue to run beyond the time limit given it by --max-runtime-hours flag.

  • Conversion to UTF encoding of client-sent logs encoded in UTF16-LE failing in some cases.

  • Current silo would not be queryable, if the lced daemon had previously been abruptly shutdown during a particular point in the silo roll sequence.

  • Changes made in WebUI to the HTTP_proxy__port configuration attribute would not be persisted.

  • Certain user-defined policies were unable to be deleted from the user interface.

  • For several particular showids command types, the lce_queryd daemon would not fill in time-range upper bound if missing.

  • Database index on silo tables' rawlog column not used unless configuration attribute position_sensitive_text_search set to true.

  • Given a showids command with multiple positive filters on the same normalized dimension, such that some but not all of those filters specified dimension literals not in DB, the lce_queryd daemon would return 0 records.

  • Restore of an archived silo snapshot could be incomplete, given mixed-case dimension literals.

Security Enhancements

  • Added HTTP Content-Security-Policy header.

  • Upgraded set of available cipher suites to prefer most secure and fall back on SHA-1 only for browsers that support nothing else.

  • Upgraded Bootstrap to prevent cross-site scripting vulnerability.

  • Upgraded jQuery and jQuery UI to prevent cross-site scripting and file upload vulnerabilities.

  • Added HTTP Cache-Control header to prevent sensitive information from being cached by the user's web browsers.

  • Upgraded Moment.js to prevent regular expression denial of service vulnerability.

  • In publicly available URL, hide version numbers if not logged in.

  • Pass session token only in server-side cookie, never in the URL.

  • Added HTTP Strict-Transport-Security header.

  • Added HTTP X-Content-Type-Options header.

  • Added HTTP X-XSS-Protection header.

  • Removed all client-side (JavaScript) cookie references.

Upgrade Notes

  • If you are upgrading from a version earlier than LCE 4.8.4, upgrade to LCE 4.8.4 before upgrading to LCE 6.0.4.
  • If you are upgrading from LCE 4.8.4 with high availability configured, use the method described in Migrate Your High Availability Configuration to LCE 6.0.4 or Later in the LCE User Guide.
  • If you are upgrading from LCE 5.0.x, upgrade to LCE 5.1.1 before upgrading to LCE 6.0.4.

  • If you are upgrading from LCE 5.x.y or 6.0.0, run:

    rpm --nopreun -Uvh lce-6.0.4-el6.x86_64.rpm

  • If you are upgrading from LCE 6.0.x to LCE 6.0.4, after the rpm command completes, run:

    nohup /opt/lce/tmp/upgr603-rebuild-silos &

    This command rebuilds your pre-existing event silos in the new format (which takes up less disk space and improves query performance). As each silo is rebuilt, it will automatically become available for querying again. The upgr603-rebuild-silos script will take 25-30 minutes to rebuild each pre-existing silo; it prioritizes silos with the most recent events.

    Supported Platforms

    • Red Hat Enterprise Linux 6 64-bit / CentOS 6 64-bit
    • Red Hat Enterprise Linux 7 64-bit / CentOS 7 64-bit