If you want to use self-signed certificates for the Tenable.sc-Nessus connection, you can perform manual Nessus SSL certificate exchange.
Caution: Please note that users should be familiar with PKI deployments and it is not recommended that the Nessus server be used as the site’s PKI system. The method described here is intended to assist in testing the functionality of the certificate exchange to assist users in the incorporation of the certificates into their current PKI system. In this method, the same key is shared between multiple servers. This may not be acceptable in some installations.