Log Correlation Engine 6.0.3 Release Notes (02-04-2020)

New Features

  • Query filter on log text may now specify that multiple tokens must be immediately adjacent in a particular order (examples follow). Token-adjacent search is disabled by default; enabling it costs a 10% to 15% increase in disk space needed for database indexes on event log text.

    • Example: text='nodeA success' will match both "nodeA success then nodeB failure" and "nodeA failure then nodeB success", just as in previous LCE releases. However, text='"nodeA success"' will match "nodeA success then nodeB failure" only; it will not match "nodeA failure then nodeB success."

    • Example: text='NodeA Success' is like grep NodeA file.txt | grep Success, whereas text='"NodeA Success"' is like grep 'NodeA Success' file.txt.

  • The v__events view now has several additional columns, to permit fine-grained visibility into event normalization. These are:

    • prm_id: identifies the exact PRM rule which had normalized this event.

    • prm_file: filename (.prm) of the plugin containing the PRM rule which had normalized this event.

    • tasl_file: filename (.tasl or .nbin) of the TASL plugin which generated this event.

  • The v__events view's kind column will now show stats, for events which had been generated by the stats daemon. This makes stats troubleshooting easier and faster.

  • For every alert created, LCE Server now stores a corresponding occasion code: cannot_DNS_resolve, client__too_long_inactive, license_expired, silo_archival_error, etc. So you can now effectively summarize recent LCE activity, with help of these new scripts under /opt/lce/tools/pg-helper-sql:

    • recent-alerts-24hours.sql: show alert counts by occasion, grouped by hour for the past 24 hours; hours without alerts are omitted; alert occasions with zero occurrences are omitted. Great for on-the-spot checks.

    • recent-alerts-14days-list.sql: show alert counts by occasion, grouped by day for the past 14 days; days without alerts are shown; occasions with zero occurrences are shown. Excellent for comparing behavior of multiple LCE instances, or of the same LCE instance over successive weeks.

  • Further about alert occasions: the new disabled_alert_occasions configuration attribute (not accessible from web UI for now) lets you specify occasions of alerts you do not want created; this lets you curtail "alert spam" proactively.

  • New script ts-stats.sql, under /opt/lce/tools/pg-helper-sql, is intended to aid troubleshooting of text search (TS) functionality. It will help you figure out what custom stopwords to add, in order to decrease disk space needed for database indexes on event log text without impairing the queries you want.

Changed Functionality and Performance Enhancements

  • As a result of switching to a more compact column representation, database silo tables will use 3% to 5% less disk space (versus LCE 6.0.0-6.0.2, for an equivalent number of events).
  • With update of the embedded events datastore from PostgreSQL 11 to PostgreSQL 12, database indexes will use 20% to 40% less disk space (versus LCE 6.0.0-6.0.2, for an equivalent number of events).
  • Write overhead has been decreased, and query performance has been improved, for many use scenarios.
  • The reset-login-account utility has been renamed to reset-account; usage is unchanged.
  • The lce_cfg_utils utility has been replaced with cfg-utils, offering far superior argument validation and error reporting; invoke with --help to see the legal usage.
  • The configure-pgServer-forPerformance script now also takes into account how much swap space has been made available to the Linux OS; this lets us allocate RAM more conservatively at sites with low swap configurations, and thus avoid out-of-memory (OOM) aborts.

Bug Fixes

  • The lce_wwwd daemon could crash, reporting segmentation fault, while shutting down.
  • Shutdown of the lced daemon could hang, if TCP syslog forwarding and/or cloud vulnerability report upload were configured.
  • Query results were not sorted in alphabetical order as per the --sortevent, --sortsen, --sorttype, or --sortuser query modifiers.
  • In cases of high log influx volume, silo rolling would fail to complete promptly, resulting in oversize silos.
  • Spurious "events-vs-rollup discrepancy" error messages appeared in lced daemon's tracelog.
  • The lced daemon would, in certain exceptional circumstances, crash right after completing a silo roll.
  • Some +text='...' query predicates were not translated to SQL predicates correctly.
  • Could not edit already-created client assignment rules from web UI.
  • Failed to persist changes to some settings configurable in Config > Advanced page of web UI.
  • Evaluation of client assignment rules did not respect the special 0.0.0.0/0 pseudo-subnet.
  • Policies with different filenames but exact same content (e.g. cloned or copied policies) were rejected.
  • The throughput--kilo-eps.sql helper script would provide only partial results, if selected time range straddled a DST calendar boundary.
  • Tenable Security Center's Detailed Event Summary tool would fail given some events normalized from Fortigate logs.
  • In some rare cases, optimize-datastore utility would not re-attach silo partitions.
  • In certain rare cases, the lced daemon would temporarily stop persisting new events.

Supported Platforms

  • Red Hat Enterprise Linux 6 64-bit / CentOS 6 64-bit
  • Red Hat Enterprise Linux 7 64-bit / CentOS 7 64-bit

Upgrade Notes

  • If you are upgrading from LCE 5.0.x, you must upgrade to LCE 5.1.1 before upgrading to LCE 6.0.3.

  • If you are upgrading from LCE 6.0.0 or LCE 6.0.1 directly to LCE 6.0.3, before upgrading to 6.0.3, run:

    service postgresql start

    source /opt/lce/tools/exigent-sessions.bashrc

    pgConf=/opt/lce/db/postgresql/postgresql.conf

    grep -q '^[ \t]*temp_tablespaces\>' $pgConf && sed -i '/^[ \t]*'temp_tablespaces'\>/ d' $pgConf && su-psqlc "DROP TABLESPACE IF EXISTS sortspace"

    Tip: If you moved your database directory, replace /opt/lce/db with the correct path.

  • If you are upgrading from LCE 6.0.x to LCE 6.0.3, after the rpm command completes, run:

    nohup /opt/lce/tmp/upgr603-rebuild-silos &

    This command rebuilds your pre-existing event silos in the new format (which takes up less disk space and improves query performance). As each silo is rebuilt, it will automatically become available for querying again. The upgr603-rebuild-silos script will take 25-30 minutes to rebuild each pre-existing silo; it prioritizes silos with the most recent events.