Most Linux distributions (e.g., Ubuntu, Red Hat, Debian) include the logrotate utility by default. Throughout this section, we will examine the standard configurations available in Ubuntu 22.04 LTS, although these options can be used in most other distributions as well. We will also discuss best practices for setting up logrotate in your environment, including:
— using create and copytruncate modes for managing log rotation
— adjusting the log rotation schedule
— compressing files to save disk space
— running and scheduling scripts for additional processing
— modifying permissions
— adding timestamps to logs
— debugging issues with the logrotate utility
— configuring environment resources to log at the appropriate level
Any Linux distribution logs all system events, such as logins, errors, and user activity, in the /var/log/ directory. In order to manage system logs, Logrotate offers two options:
- For applying rotation settings globally, a configuration file may be found at /etc/logrotate.conf
- For configuring log rotation per package or service installed on the system (e.g., mysql-server, apache), there is a directory named /etc/logrotate.d.
shell> nano /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
Create or copy log files to manage rotation
Logrotate provides two primary directives for specifying how new and existing files are rotated: create and copytruncate. The functionality of the application is checked against the requirements of the end user during acceptance testing. This concludes the testing process. When the end user is satisfied with the application’s features and functionalities, it is moved to the production environment.
In contrast, the copytruncate directive instructs logrotate to first create a copy of the original log file and then delete its contents in order to reduce its size (i.e., truncate the file). Due to the fact that it accommodates continuous, uninterrupted writing to the original file, this behavior is less disruptive than create mode. By copying the file (e.g,. /var/log/apache2/error.log.1) and then deleting the data from the original one rather than renaming it-logrotate will ensure that the logging service can continue to write to the file without interruption.
In the case of a service that cannot close and reopen log files without restarting, copytruncate mode may be useful. It is possible that system daemons and other background processes are not configured to handle termination signals such as SIGHUP in order to reopen log files. Although this mode consumes a greater amount of resources than create mode, it can result in race conditions that may result in data loss. It is possible, for example, that any data logged after the copy operation but prior to the truncate operation may be lost. For most applications, it is recommended to use the create mode due to these caveats.
Adjust your log rotation schedule
With Logrotate, you can automatically rotate files on a daily, weekly, monthly, or yearly basis-or when the file size reaches a certain threshold. Rotating files on a regular basis is an appropriate solution for services that generate a consistent volume of logs. To ensure that disk space is not consumed excessively, services that experience periodic spikes in activity or generate large volumes of log files may require size limits.
Logrotate offers the following options for controlling your size-based rotation:
— size: rotate by a specified size on a daily basis (overrides default schedule)
— minsize: rotate based on time interval but only when files are at least this size
— maxsize: rotate when files exceed this size, even before the time interval
shell> nano /etc/logrotate.d/apache2
/var/log/apache2/*.log {
weekly
maxsize 1G
}
Modify permissions to grant access to logs
In the default configuration, only the root user in the adm group has access to the /var/log/ directory. As a result of this setting, logrotate is permitted to modify most logs, however there may be situations in which you are required to update your settings in order to grant additional access. Depending on the version of MySQL, a database server running on Linux systems, the database logs may be accessed by only the mysql user and group. This can be accomplished by updating the adm group in the associated configuration file of the mysql-server service from mysql to logrotate’s adm group, as shown below:
shell> nano /etc/logrotate.d/mysql-service
/var/log/mysql.log /var/log/mysql/*log {
[...]
create 640 mysql adm
[...]
}
The utility will have access to the database log files (e.g., /var/log/mysql/mysql_error.log) in order to successfully manage rotation, while still allowing the mysql-service service to write new log files.
In the Datadog Agent, the dd-agent user and group are responsible for running the software. Using an ACL-enabled environment, you can grant the Agent access to a log directory, such as /var/log/apache:
Shell> setfacl -m u:dd-agent:rx /var/log/apache2
The above command uses the -rx option to give the Agent user read and execute permissions for Apache logs.
You should be aware that the above command applies the ACL setting only once, so you must ensure that the configuration persists for rotated logs. It is possible to do this by defining the logs that you would like Datadog to forward to the Agent in the dd-agent_ACL file in the /etc/logrotate.d directory.
shell> /etc/logrotate.d/dd-agent_ACL
{
postrotate
/usr/bin/setfacl -m g:dd-agent:rx /var/log/apache2/error.log
/usr/bin/setfacl -m g:dd-agent:rx /var/log/apache2/access.log
endscript
}
