Puppet Managed Nodes Logs

This section explains how logs are collected from Puppet-managed nodes using the Monit Agent and how you can view and customize log collection. By default, only system logs from the Syslog are collected, providing essential information about system events and issues. These logs are forwarded to CERN’s monitoring infrastructure, where they can be accessed for centralized analysis and troubleshooting.

Getting started

By default, the Monit Agent is configured to collect logs exclusively from the Syslog service. This setup captures logs from system services and applications that write to the system log, giving insight into important events on your node. All collected Syslog logs are forwarded to the central logging endpoint at monit-timberprivate, where you can view and analyze them.

To get started:

Verify Log Collection: Ensure that your application or service is logging relevant events to the system's Syslog, as only these logs are gathered by the default Monit Agent setup. By default only program names in the monitoring::monit_agent::syslog_whitelist list are allowed to being sent, but you can adapt it as needed using Hiera.
Access Logs: Go to monit-timberprivate.cern.ch to view logs from your node. You can search, filter, and analyze logs for insights into system and application behavior (Make sure you are under your tenant scope).

How do I send logs from my application?

If your application logs to a custom file or another logging system (outside of Syslog), you’ll need to set up additional configurations to collect these logs through the Monit infrastructure. By default, only Syslog logs are gathered, so custom logs require a separate setup.

To send custom logs from your application:

Instantiate a New Fluent Bit Instance: Use Puppet to deploy and configure a new instance of Fluent Bit on the node. This additional Fluent Bit instance will handle custom log files or other log sources not covered by Syslog. In order to easy the set up install Fluent Bit using the Monit Agent wrapper.

When enabling the new agent, it will also enable the new MONIT repositories in the machine, which contain the package "monit-fluent-bit".

Although it has a different name, it's been built as a virtual package for fluent-bit, so in case you were using it before the new package will be identified as an upgrade.

This by itself should not be a big issue (unless some breaking change between versions), but the problem might come with the way the new package service works. This new package is shipped with a service enabled to run multiple instances of fluentbit in a single machine via "fluent-bit@".

So you will need to adapt your current configuration to work with this, there are few ways of managing it, but we recommend using this wrapper provided by us.
```
# Install fluentbit
include monitoring::monit_agent::forwarders::fluentbit::install 

# Fluent-bit configuration constants.
$fluentbit_agent_name = 'my-agent'
$fluentbit_service_name = "fluent-bit@${fluentbit_agent_name}.service"
$fluentbit_agent_config_base_dir = "/etc/fluent-bit/${fluentbit_agent_name}"

# Instantiate a fluent-bit service as monit-agent
monitoring::monit_agent::forwarders::fluentbit::agent { $fluentbit_agent_name: }
```
This will make sure you have your service environment configuration fluent-bit@my-agent and all the folders needed where to place configuration.

Configure Custom Log Scraping: Set up the new Fluent Bit instance to scrape logs from your desired log file or directory. Configure it to forward these logs to the Monit infrastructure, ensuring they are sent to the correct endpoint for centralized analysis.

# My Fluent Bit input
fluentbit::input::tail { 'my-application-fluentbit-input':
    configfile        => "${fluentbit_agent_config_base_dir}/my-application-fluentbit-input.conf",
    path              => '/var/log/my-application/*.log',
    routing_tag       => 'my-application-logs',
    multiline_parsers => [
    'java_multiline',
    ],
}

# My Fluent Bit output to monit opentelementry
fluentbit::output::opentelemetry { "my-application-fluentbit-output":
    configfile   => "${fluentbit_agent_config_base_dir}/my-application-fluentbit-output.conf",
    host         => 'monit-otlp.cern.ch',
    port         => 4318,
    metrics_uri  => '/v1/metrics',
    logs_uri     => '/v1/logs',
    traces_uri   => '/v1/traces',
    match        => 'my-application-logs',
    headers      => {
        tenant_id => 'your-tenant',
    },
    service_name => $fluentbit_service_name,
}

Verify Log Forwarding: Once configured, test that your custom logs are being forwarded correctly to monit-timberprivate.cern.ch by checking for the new log entries in the system. If new logs do not appear them you can try to debug your Fluent Bit installation and configuration. For that, check the following: 1. Your Fluent Bit installation appears in the host. 2. Your Fluent Bit configuration files are where they are supposed to be. 3. Your Fluent Bit is loading the config from where it is supposed to do so. 4. Check for errors in the logs of your Fluent Bit installation. If the error persists please, open a snow ticket to the monitoring service.

How can I see my logs?

Logs are essential for monitoring and troubleshooting applications, providing insights into system performance, errors, and usage patterns. At CERN, logs are collected and centralized to ensure seamless access and analysis across multiple platforms. You can view and manage your logs using OpenSearch for detailed searches and Grafana for visual dashboards. Below are the steps to access and utilize these tools effectively.

Viewing Logs in OpenSearch

All logs collected by the Monit Agent, including those from Syslog and custom sources, are stored and accessible in CERN's centralized logging platform, OpenSearch, at monit-timberprivate.cern.ch. To view your logs:

Access the OpenSearch Platform:
Navigate to monit-timberprivate.cern.ch in your browser. You’ll need to authenticate using your CERN account. Once logged in:
Go to the Discover view.
Locate the index pattern assigned to your tenant. If you're unsure about your index pattern or can't find it, open a SNOW ticket for assistance.
Search for Your Node’s Logs:
Use the search functionality to find logs for your specific node or application. You can filter by:
Timestamp
Log Level
Specific Keywords
Analyze and Set Alerts:
OpenSearch allows you to analyze log patterns and set alerts based on specific criteria, such as error levels or recurring issues. This proactive approach helps monitor application health and detect issues early.

Custom Fluent Bit logs, if properly forwarded, will also appear here for unified log management across both Syslog and custom application logs.

Viewing Logs in Grafana

For a graphical representation of your logs and metrics, you can use Grafana, available at monit-grafana.cern.ch. Here’s how:

Access the Grafana Platform:
Go to monit-grafana.cern.ch and log in using your CERN account.
Create the Data Source:
To view your logs in Grafana, you need to configure an OpenSearch data source.
Navigate to Configuration > Data Sources in Grafana.
Click Add Data Source and select OpenSearch as the type.
Provide the following details:
- URL: https://monit-timberprivate.cern.ch
- Authentication: Use your tenant name as the username and your tenant password as the password.
- Index Pattern: Set the index pattern to monit-otel-logs_<tenant_name>.
- Click Save & Test to verify the connection.
Explore Dashboards:
In Grafana, you can view preconfigured dashboards that provide insights into your logs, metrics, and application performance.
Use the search bar to find relevant dashboards.
Customize or filter the visualizations to focus on specific logs, time ranges, or patterns.
Set Alerts and Notifications:
Grafana enables you to configure alerts for specific log patterns or thresholds. Set up notifications via email or other communication channels to stay informed of critical issues.