Tenable Log Correlation Engine 6.0.10 Release Notes (03-17-2022)

Note: If you are upgrading from any version earlier than 6.0.0, upgrade directly to 6.0.10. See Upgrade Notes for detailed information about supported upgrade paths.

New Features

• Now also available for OEL8 / RHEL8 / CentOS8.

• New scripts under /opt/lce/tools/pg-helper-sql/:

  • all-columns--decl-order.sql [<table name fragment>]

    • Useful when specifying PostgreSQL statistics objects.

  • planner-estimates.sql <table name> [<mCommonestValues>,=10]

  • progress--analyze.sql [<refreshInterval_seconds>,=15]

  • progress--bulk-load.sql [<refreshInterval_seconds>,=5]

    • Useful for performance tuning of the lced daemon's [a] persisting of events to activeDb, and [b] saving silo snapshots into archiveDb.

  • progress--stream-backup.sql [<refreshInterval_seconds>,=60]

  • rawlog-storage.sql <silo#> <percent of rows to scan>

  • table-buffer-cache-stats.sql [<table name fragment>]

• New shell function list-helper-SQL-scripts-with-usage, which becomes available in a shell session after you have once run source /opt/lce/tools/exigent-sessions.bashrc in that session; it lists the name and usage banner of every helper .sql script bundled and invocable with psqlf or psqli, as explained in the Tenable Log Correlation Engine User Guide.

• New action --recompress now afforded by the archival-manager utility; this retroactively changes the gzip compression level of the constituent files of an archived snapshot in the original location.

• Additional optional arguments, -<N_newest> and <N_oldest> are now accepted by the archival-manager utility's --list-snapshots action.

• Additional counters tracked by the lce_queryd daemon let you quickly identify showids command (encoded query sent from Tenable Security Center) problem areas:

  • The riskSlow group, counting queries which likely take significant time to evaluate.

  • The foreknow_noSuch group, counting queries which could never return any results.

• The diag utility now takes an optional --sanitize argument. If this is given, the diagnostics report produced will have overwritten the following: IPv4 addresses, MAC addresses, CPU serial numbers, chassis serial numbers, and motherboard serial numbers.

• New sections in diag report:

  • operational-logs/querying-profile.txt

    • For each query received by Log Correlation Engine in the most recent few months, prints a line with:

      • response latency in seconds, or timed_out

      • abbreviated name of Tenable Security Center tool

      • query time span

      • traffic direction

      • whether this was an “archive-peek” query

      • which predicates were used.

    • Because lines are formatted consistently, this report is easy to analyze with standard UNIX utilties, a scripting language, or a spreadsheet program.

  • operational-logs/rawlog-queries-distribution.txt

    • Reports how many times, in the most recent few months, has Log Correlation Engine received a query containing a particular +text filter (in other words, a rawlog predicate). Since such queries are often among the slowest to execute, changing them to filter instead by normalized attributes can significantly improve your Tenable Security Center Events Analysis experience.

  • ssl_config.txt

    • Lists the names and sizes of SSL credential files found, grouped by containing directory.

• New sub-sections in diag report:

  • In the clients.txt section, “Nonunique sensors”.

  • In the disk__space_inodes_mounts.txt section, “LVM: physical volumes … volume groups … logical volumes”.

• SSL credentials for Web UI may now be rotated independently of SSL credentials for vuln reporter proxy: the former with lce_crypto_utils --generate-creds-webUI, and the latter with lce_crypto_utils --generate-creds-vulnReporter.

Changed Functionality and Performance Enhancements

• Underlying datastore upgraded from PostgreSQL 12.1 to PostgreSQL 14.1; as result, Log Correlation Engine realizes the following benefits:

  • Reduced event persistence overhead, due to faster compression of long rawlog strings and to improved performance of binary-mode COPY FROM.

  • Less disk needed per silo, due to more space-efficient storage of duplicate entries in BTree-type indexes on low-cardinality columns. (The upgr6010-partial-reindex-silos utility will rebuild indexes on silos created prior to upgrade, so you can take advantage of this improvement retroactively.)

  • Improved querying performance in many scenarios.

  • Less interference with daemons' operation by the optimize-datastore utility, due to VACUUM command operation being now parallelizable.

• Speedup of up to 9%, subject to available host computing resources, of:

  • Saving of silo snapshots into archiveDb by the lced daemon.

  • All "archive-peek" operations by the queryd daemon.

• The queryd daemon now recognizes and removes the " mark that Tenable Security Center occasionally appends to the showids queries it sends to Log Correlation Engine, resulting in an illegal query which Log Correlation Engine used to (rightly) reject. This is a workaround for a Tenable Security Center bug.

• The queryd daemon now recognizes and removes the “+event *_* +event *-*” filter sequence that Tenable Security Center frequently inserts into the showids queries it send to Log Correlation Engine, resulting in gratuitous use of computer resources and a needless slowdown.

• Added the "inverse meepfile" optimization to queryd daemon; when applicable, this makes IP address filters smaller and hence conducive to more efficient evaluation.

• Added the "inline specification of IP address filters" optimization to queryd daemon; this lets PostgreSQL take advantage more often of indexes on src_ip and/or dst_ip columns, with resultant query execution speedup.

• To prevent transition into an invalid configuration when adding or editing a client assignment rule, Log Correlation Engine now checks, for every policy P mentioned in said rule, that P exists and has been successfully loaded.

• To prevent transition into an invalid configuration when adding or editing the include_networks or exclude_networks configuration attribute, Log Correlation Engine now checks that no include_networks subnet overlaps with an exclude_networks subnet and vice versa.

• To prevent transition into an invalid configuration when deleting a policy P, Log Correlation Engine now checks that P is not referenced by any client assignment rules.

• The make_cert utility has been removed, and its functionality moved into lce_crypto_utils.

• Renamed scripts under /opt/lce/tools/pg-helper-sql/:

  • progress--index-or-reindex.sql, previously command-progress--index--create-or-rebuild.sql

  • progress--rebuild-table.sql, previously command-progress--cluster.sql

  • progress--vacuum.sql, previously command-progress--vacuum.sql

  • planner-estimates--silo.sql, previously planner-estimate-basis.sql

Bug Fixes

• The lced daemon could enter dynamic halt during silo roll.

• The lced could fail to complete a silo roll; in such a scenario, lced would continue to persist events to the last-added silo, but would not perform subsequent silo rolls, trim activeDb as needed, or trim archiveDb as needed.

• The tasld and statsd daemons could enter dynamic halt if either began working on a silo S, and S was aged out of activeDb before the daemon in question finished working on S.

• The lced daemon could fail to persist updated client activity counters, such as how many events received in the current day or exactly when had an event been last received.

• The RPM installer/upgrader did not halt optimize-datastore process if one was running.

• The diag utility did not correctly report disk space statistics for the filesystem containing a site's archiveDb, if said filesystem was a network mount.

• The lced daemon, in certain rare cases at high-volume site, would fail to persist a minute fraction of events accepted just prior to silo roll.

• Authorization of encrypted syslog senders via the crypt_syslog__authorized_fingerprints configuration attribute did not work in some cases.

• In some cases, archival peek status would not be properly carried across reboots of the queryd daemon.

• An Log Correlation Engine client's IP address and/or sensor name would not be updated in the Log Correlation Engine Server database immediately after the IP address of the client's host changed due to a DHCP re-assignment; the desired update would only happen after a disconnect/reconnect sequence.

• License activation and plugins update failing after January 2022, due to heightened SSL protocol level requirements.

• The queryd daemon would fail to generate response to a showids command requesting some attributes of every event in specified timerange (without any rollup or summarization), if no response size limit had been specified.

• An instance of showids launched by Tenable Security Center could fail to exit while holding on to a PostgreSQL connection; if this happened often enough, PostgreSQL connections would be exhausted, and no utilities would function normally.

• In some cases, the cfg-utils utility, when issued a command with an invalid argument, would emit an error message stating that PostgreSQL datastore is unavailable.

• The --set-sv action of the cfg-utils utility, when used to set activeDb_max_on_disk_size__MB or archiveDb_max_on_disk_size__MB configuration attribute, would interpret its numeric argument as denoting bytes instead of megabytes.

• Credential files produced by the lce_crypto_utils utility did not allow for browser access.

Security Enhancements

• Updated to OpenSSL 1.1.1l

• Prevent a Cross-site Scripting (XSS) vulnerability in the DataTables plugin.

• Prevent an arbitrary code injection via the template function in Underscore plugin.

• Prevent jQuery-UI from executing untrusted code.

• Prevent Regular Expression Denial of Service (ReDoS) in CodeMirror plugin.

• Prevent Remote Code Execution (RCE) in Handlebars compiler on templates coming from an untrusted source.

Upgrade Notes

• If you are upgrading from a version earlier than Log Correlation Engine 4.8.4, upgrade to Log Correlation Engine 4.8.4 before upgrading to Log Correlation Engine 6.0.10.
• If you are upgrading from Log Correlation Engine 5.0.x, upgrade to Log Correlation Engine 5.1.1 before upgrading to Log Correlation Engine 6.0.10.
• If you are upgrading from Log Correlation Engine 4.8.4 with high availability configured, use the method described in Migrate Your High Availability Configuration to LCE 6.0.4 or Later in the Tenable Log Correlation Engine User Guide.
• If you are upgrading from 6.0.4 with high availability configured, use the method described in Migrate Your High Availability Configuration to LCE 6.0.5 or Later in the Tenable Log Correlation Engine User Guide.

• If you are upgrading from Log Correlation Engine 5.x.y or 6.0.0, run:

  rpm --nopreun -Uvh lce-6.0.10-el6.x86_64.rpm

• If you are upgrading from Log Correlation Engine 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, or 6.0.5, after the rpm command completes, run:

  nohup /opt/lce/tmp/upgr606-rebuild-hhourlies &

  Note: The upgr606-rebuild-hhourlies script takes approximately 2.5 minutes to run per activeDb silo.

• If you are upgrading from Log Correlation Engine 6.0, 6.0.1, or 6.0.2, after the rpm command completes, run:

  nohup /opt/lce/tmp/upgr603-rebuild-silos &

  This command rebuilds your pre-existing event silos in the new format (which takes up less disk space and improves query performance). As each silo is rebuilt, it will automatically become available for querying again. The upgr603-rebuild-silos script will take 25-30 minutes to rebuild each pre-existing silo; it prioritizes silos with the most recent events.

  Note: If you are upgrading from Log Correlation Engine 6.0, 6.0.1, or 6.0.2, run /opt/lce/tmp/upgr606-rebuild-hhourlies first, then upgr603-rebuild-silos.

• If you are upgrading from Log Correlation Engine 4.8.4, 5.1.1, 6.0, 6.0.1, 6.0.2, or 6.0.3, you may be prompted to run /opt/lce/tmp/restore_per-client_decisions.sql when the upgrade utility completes. If you receive this prompt and you performed explicit client authorizations (without the aid of client assignment rules or the auto-authorization setting) or specified custom sensor names for individual Log Correlation Engine clients, run:

  source /opt/lce/tools/source-for-psql-shortcuts.sh

  psqlf /opt/lce/tmp/restore_per-client_decisions.sql

  This script applies the changes you made to your upgraded Log Correlation Engine.

  Note: If you did not perform explicit client authorizations or specify custom sensor names for individual Log Correlation Engine clients, you do not need to run /opt/lce/tmp/restore_per-client_decisions.sql

• (Optional) If you are upgrading from Log Correlation Engine 6.0.0-6.0.9, after the rpm command completes, run the following command to recover disk space:

  sleep 5h ; nohup /opt/lce/tmp/upgr6010-partial-reindex-silos &

  Note: This script may require up to 20 minutes per activeDb silo to run.

  Note: If you are upgrading from Log Correlation Engine 6.0, 6.0.1, or 6.0.2, run /opt/lce/tmp/upgr606-rebuild-hhourlies first, then upgr603-rebuild-silos, then upgr6010-partial-reindex-silos.

  Note: If you are upgrading from Log Correlation Engine 6.0.3, 6.0.4, or 6.0.4, run /opt/lce/tmp/upgr606-rebuild-hhourlies first, then upgr6010-partial-reindex-silos.

Supported Platforms

• Red Hat Enterprise Linux 6 64-bit / CentOS 6 64-bit

• Red Hat Enterprise Linux 7 64-bit / CentOS 7 64-bit

• Red Hat Enterprise Linux 8 64-bit / Oracle Enterprise Linux 8 64-bit