You are here: Features > Configuration > Advanced Configuration > Debugging

Debugging

If any changes are made to the Debugging section below, select the Update button at the bottom of the Advanced page for the changes to go into effect.

Option Description
Write Unnormalized Logs

If this is enabled, LCE will create a file named notmatched.txt in the database directory and fill it with log events that have not matched any LCE plugin. This is an excellent way to analyze events that may be inadvertently ignored. There is a hardcoded limit of 2 GB for this option in addition to the number of events specified.

Caution: This option is deprecated - users are encouraged to instead enable “Store Unnormalized Logs” above. If non-zero, this is the number of unnormalized logs to write to the rolling notmatched.txt file in the database directory.

Save All Logs File

Specifics a log file where all events (not just the ones matched with a LCE plugin) are stored. This log file does not rotate and must be managed by the logrotate process. Note that this will require significantly more disk space than just keeping the events that match plugin criteria. This option is most useful when used in conjunction with logrotate and an external storage device.

Caution: Deprecated - this should be enabled temporarily for debugging only. The “Save All Logs File” option is only useful if a text version of all incoming logs is desired.

Enable Multiple Matches

By default, LCE stops evaluating plugins when it encounters a match for a log. If this option is enabled, LCE will evaluate all plugins for each log.

Log Client Event Packets

LCE server receives an event or event-related message from a LCE Client.

Log Client Authorization

LCE server receives a login, logout, version info, or related message from a LCE Client.

Log Server Client Tracking

LCE server connects, disconnects, updates status for, or performs related actions for a LCE Client.

Log Plugin Matches (successful)

LCE server successfully matches a log with a plugin match statement.

Log Plugin Matches (failed)

LCE server fails to match a log with a plugin match statement.

Log Plugin Matches (attempted)

LCE server attempts to match a log with a plugin match statement.

Log Plugin Construction

LCE server parses the plugins and constructs internal representations

Log Plugin Match Organization

LCE server sorts and builds the plugin execution structure internally.

Log Silo Rollover

LCE server fills a silo and prepares to write to the subsequent silo.

Log Load Balanced Data

LCE server offloads an event to an Auxiliary LCE, or LCE server receives an event from Primary LCE in a load balancing configuration.

Log Load Balanced Status

LCE server receives a status heartbeat from an Auxiliary LCE, or LCE server sends a status heartbeat to the Primary LCE in a load balancing configuration.

Log Load Balance Connections

LCE server connects or disconnects to another LCE in a load balancing configuration

Log High Availability

LCE server connects, disconnects, fails over, or performs a related action in high availability mode.

Log Reconfiguration

LCE server receives a configuration update from the web-based user interface.

Log User Tracking

LCE server processes an event with a normalized user name and performs a user tracking action.

Debug Mode

It is possible to add various types of debug parameters in the LCE GUI. Information about plugins loaded, LCE client status, and operation can all be written to the current log file.

The LCE GUI Debugging section can be used to log all remote client authentication attempts by enabling Log Client Authorization, which can be helpful when diagnosing remote agent problems. One activity that can be logged is the Log Silo Rollover, which logs when a silo is rotated and indexed.

Note: Enabling these debug messages is a great way to learn how the LCE operates and troubleshoot issues. However, they can generate a lot of information and can create multi-gigabyte log files when left enabled.

If the lced daemon terminates abnormally for any reason, the system will automatically restart the daemon and add a warning to the LCE logs.

Storing All Logs with “save-all”

Many organizations have regulatory requirements to save all of their log data for a specified length of time. It may also be part of that requirement that the data not be manipulated, normalized, or otherwise processed in case it must be used in a legal proceeding. Any exculpatory evidence in the original logs must not be missing as well.

The LCE’s method of storing data in silos for high-speed normalization and analysis by many different administrators is not the best place to keep one central log file. The LCE has means to save every message, even ones that do not match a certain plugin to a central log file.

This log file can be saved by adding the full path to the log file under the Save All Logs File section found in the Debugging section of the advanced menu in the LCE GUI. The default location of the Save All Log File in previous versions of LCE was /opt/lce/db/lce.log, which is in the same directory as the silos, but it can be changed to any desired location that has adequate disk space. In new installations, the path and filename must be specified.

As the LCE daemon receives events through the API or from syslog, it will save the message into the file specified in the LCE GUI. This log file will grow very large. Maintain rotation and compression of these logs with the logrotate program that is already installed on all Linux systems supported by the LCE.

Different File System

Since the file that stores all the log files will grow to extremely large sizes when left enabled, it is highly recommended to place this file on a different physical file system. If the LCE server is placed on a system with two hard drives, consider creating physically separate partitions for both the LCE silo data and the “save-all” files.

If your network has use of a Storage Area Network (SAN), consider using this to store the “save-all” file. Many times, these storage devices can be mounted through a network file system (NFS) or Windows file share (SMB) resource. Make sure that write permissions from the LCE server are available and there is sufficient network bandwidth to send the data, if you use a SAN.

Multiple Plugin Matches per Log File “multiple-matches”

By default, the LCE daemon will stop processing a log file as soon as one match has been made. This behavior may be overridden by selecting “Enable Multiple Matches” in the Debugging section of the Advanced menu in the LCE GUI. With this feature enabled, the LCE daemon will attempt to exercise the entire plugin set across every log message. This behavior is useful for extracting multiple forms of information out of a log file. For example, there may be a plugin that looks for a generic user login failure and another that looks for a login failure for user “root”. Without the multiple matches option enabled, only one of the plugins will match, even though both are valid.

Note: Even more so than with normal LCE operation, be sure to remove unneeded libraries with multiple matches feature enabled, otherwise the LCE’s performance can be diminished.

Quick Example

This examples supposes an organization that has a firewall log with NAT addresses. For each transaction, the firewall logs the external Internet address, the organization’s Internet address and their internal RFC1918 address. The organization wants the ability to type in any of the IP addresses in question to produce a report of the history.

Suppose that a student is assigned 192.168.20.10 via DHCP inside a high school. The school’s public IP address at the firewall may be 64.64.64.64 and the student attacks a web site at 99.99.99.99.

A firewall log may have all three IP addresses for any network browsing. Without Enable Multiple Matches options selected, there is only one pair of IP addresses that can be matched. However, with Enable Multiple Matches enabled, two rules can be used to process the same log file and extract the specific IP addresses.

In this example, the organization already monitors “external to public IP” and “public IP to internal IP” firewall logs. They generate two LCE events for each firewall log event. By adding the DHCP logs, the organization is able to use the IP address of the attacked target to get the actual internal IP address and MAC address of the host used to perform the attack.

Copyright © 2017. Tenable Network Security, Inc. All rights reserved. Tenable Network Security, Nessus, SecurityCenter Continuous View, Passive Vulnerability Scanner, and Log Correlation Engine are registered trademarks of Tenable Network Security, Inc. All other products or services are trademarks of their respective owners.