Managing Services with systemd

reddit [#143] Created with Sketch.
Venkat Vaddineni
Venkat Vaddineni
1 min read

Managing Services with systemd

When you start working with Linux servers, one of the first things you’ll encounter is the need to manage services - programs that run in the background to provide functionality like web servers, databases, or custom applications. Systemd is the modern service management system used by most Linux distributions today, and understanding it is crucial for any DevOps engineer.

Think of systemd as the traffic controller for your Linux system. It decides when services start, stop, and how they interact with each other. Unlike older systems that started services one by one in sequence, systemd can start multiple services simultaneously, making your system boot faster and run more efficiently.

What is systemd and Why Does It Matter?

Systemd replaced the older SysV init system and has become the standard across major Linux distributions like Ubuntu, CentOS, RHEL, and Debian. As a DevOps engineer, you’ll interact with systemd daily when deploying applications, troubleshooting issues, and maintaining server infrastructure.

The key advantages of systemd include:

Parallel Service Startup: Instead of waiting for each service to start completely before starting the next one, systemd can start multiple services at once, significantly reducing boot times.

Dependency Management: Systemd understands which services depend on others and ensures they start in the correct order. For example, your web application won’t try to start before the database it needs is running.

Comprehensive Logging: All service logs are centralized in one place, making troubleshooting much easier than hunting through multiple log files.

Automatic Restart Capabilities: Services can be configured to automatically restart if they crash, improving system reliability.

Resource Management: You can limit how much CPU, memory, or other resources a service can use, preventing one misbehaving service from bringing down your entire system.

Understanding systemd Units

In systemd terminology, everything it manages is called a “unit.” There are different types of units for different purposes:

Service Units are the most common type and represent the services (programs) running on your system. These end with .service and include things like web servers, databases, and your custom applications.

Target Units group related services together, similar to the old concept of runlevels. For example, the multi-user.target represents the state where the system is ready for multiple users to log in.

Timer Units are systemd’s equivalent to cron jobs, allowing you to schedule tasks to run at specific times or intervals.

Socket Units allow services to start only when someone tries to connect to them, saving system resources.

Understanding these unit types helps you grasp how systemd organizes and manages different aspects of your system.

Basic Service Management

Let’s start with the fundamental operations you’ll perform daily. The main command for interacting with systemd is systemctl.

Checking Service Status

The most common task is checking whether a service is running:

systemctl status nginx

This command shows you whether the nginx web server is active (running), how long it’s been running, and recent log messages. The output uses color coding: green usually means everything is working, red indicates problems.

You can also get a quick yes/no answer about whether a service is running:

systemctl is-active nginx

This returns either “active” or “inactive” without the detailed information.

Starting and Stopping Services

To start a service that isn’t running:

systemctl start nginx

This starts the service immediately but doesn’t configure it to start automatically when the system boots.

To stop a running service:

systemctl stop nginx

To restart a service (stop it completely, then start it again):

systemctl restart nginx

Sometimes you want to reload a service’s configuration without fully stopping it:

systemctl reload nginx

This tells the service to re-read its configuration files while continuing to run. Not all services support this operation.

Enabling Services for Boot

Starting a service manually only keeps it running until the system reboots. To make a service start automatically when the system boots:

systemctl enable nginx

This creates the necessary links so systemd knows to start this service during boot. The service isn’t started immediately - you need to use start for that.

To prevent a service from starting at boot:

systemctl disable nginx

You can check if a service is enabled for boot:

systemctl is-enabled nginx

Getting Service Information

To see all available services on your system:

systemctl list-units --type=service

This shows all services, their current state, and a brief description. You can also see services that failed to start or have other issues.

To see detailed information about a specific service:

systemctl show nginx

This displays all configuration options and current values for the service, which is useful for troubleshooting.

Understanding Service Configuration Files

Every service is defined by a configuration file called a unit file. These files tell systemd how to start the service, what user it should run as, what other services it depends on, and much more.

Where Unit Files Live

Systemd looks for unit files in several locations, with a specific priority order:

System unit files (/lib/systemd/system/ or /usr/lib/systemd/system/) contain the default configurations installed by packages. You shouldn’t modify these directly because package updates might overwrite your changes.

Local unit files (/etc/systemd/system/) are where you place custom unit files or modifications to existing ones. These take priority over system unit files.

Runtime unit files (/run/systemd/system/) are created dynamically and exist only while the system is running.

Anatomy of a Unit File

Unit files are organized into sections, each with a specific purpose. Let’s examine a typical service unit file:

The [Unit] section contains general information about the service:

[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

The Description is a human-readable explanation of what the service does. The After directive tells systemd to start this service only after the network is ready, remote filesystems are mounted, and name service lookups are working.

The [Service] section defines how the service should run:

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
User=nginx
Group=nginx

Let’s break this down:

  • Type=forking means the main process will fork (create a copy of itself) and the parent will exit, leaving the child running in background
  • PIDFile tells systemd where to find the process ID of the main service process
  • ExecStartPre runs a command before starting the main service (here, testing the nginx configuration)
  • ExecStart is the actual command to start the service
  • ExecReload defines how to reload the service configuration
  • User and Group specify which user account the service should run under

The [Install] section determines when and how the service should be enabled:

[Install]
WantedBy=multi-user.target

This means when you enable the service, it will be started as part of the multi-user target (normal system operation).

Service Types

The Type parameter in the service section is crucial for understanding how systemd manages your service:

simple (the default) means your program runs in the foreground and doesn’t fork. This is the most straightforward type for most applications.

forking means your program will start, create a background process, and then the original process exits. Traditional Unix daemons often work this way.

oneshot is for programs that run once and exit, like initialization scripts or backup jobs.

notify is for programs that can tell systemd when they’re fully ready to serve requests.

Understanding these types helps you choose the right configuration for your services.

Working with Dependencies

One of systemd’s most powerful features is its ability to manage service dependencies. This ensures services start in the right order and that required services are available when needed.

Types of Dependencies

Requires creates a strong dependency. If the required service fails to start, this service won’t start either. If the required service stops unexpectedly, this service will also be stopped.

Wants creates a weaker dependency. Systemd will try to start the wanted service, but if it fails, this service can still start.

After and Before control ordering. After=network.target means “start this service after the network is ready,” but doesn’t require the network to be working.

Practical Example

Consider a web application that needs a database:

[Unit]
Description=My Web Application
After=network.target
Wants=postgresql.service

This configuration tells systemd:

  1. Start this service after the network is ready
  2. Try to start PostgreSQL, but don’t fail if it’s not available (maybe the database is on another server)

For a more critical dependency:

[Unit]
Description=My Web Application  
After=network.target postgresql.service
Requires=postgresql.service

Now the web application requires PostgreSQL to be running and will fail to start if PostgreSQL isn’t available.

Managing Logs with journald

Systemd includes its own logging system called journald, which captures all output from services and system events in one centralized location.

Viewing Service Logs

To see logs for a specific service:

journalctl -u nginx

This shows all log entries for the nginx service. The output includes timestamps, the service name, and the actual log messages.

To follow logs in real-time (like tail -f):

journalctl -u nginx -f

This is incredibly useful for watching what happens when you test your application or troubleshoot issues.

Filtering Logs

You can filter logs by time period:

journalctl -u nginx --since today
journalctl -u nginx --since "2023-01-01 10:00:00"
journalctl -u nginx --since "1 hour ago"

You can also filter by priority level:

journalctl -u nginx -p err

This shows only error messages, filtering out informational messages that might clutter the output.

Understanding Log Priorities

Systemd uses standard syslog priority levels:

  • emerg (0): System is unusable
  • alert (1): Action must be taken immediately
  • crit (2): Critical conditions
  • err (3): Error conditions
  • warning (4): Warning conditions
  • notice (5): Normal but significant condition
  • info (6): Informational messages
  • debug (7): Debug-level messages

When troubleshooting, start with error and critical messages, then work your way down to informational messages if needed.

Creating Custom Services

As a DevOps engineer, you’ll often need to create custom service files for applications that don’t come with systemd integration.

Planning Your Service

Before creating a unit file, consider:

  • What user should the service run as? (Never run services as root unless absolutely necessary)
  • What other services does it depend on?
  • Should it restart automatically if it crashes?
  • What environment variables does it need?

Basic Custom Service Example

Let’s create a service for a simple web application:

[Unit]
Description=My Custom Web App
After=network.target

[Service]
Type=simple
User=webapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Let’s understand each part:

Unit Section:

  • Describes what the service does
  • After=network.target ensures the network is ready before starting

Service Section:

  • Type=simple because our app runs in the foreground
  • User=webapp runs the service as a dedicated user (better security)
  • WorkingDirectory sets where the process should run from
  • ExecStart is the command to start the application
  • Restart=always means systemd will restart it if it crashes
  • RestartSec=10 waits 10 seconds before restarting

Install Section:

  • WantedBy=multi-user.target means it starts during normal system operation

Implementing the Service

Save this file as /etc/systemd/system/myapp.service, then:

sudo systemctl daemon-reload

This tells systemd to scan for new or changed unit files.

sudo systemctl enable myapp
sudo systemctl start myapp

Then check that it’s working:

sudo systemctl status myapp

Security Considerations

Running services securely is crucial in production environments. Systemd provides many options to limit what services can do.

User and Group Isolation

Always run services as dedicated users with minimal privileges:

[Service]
User=myapp
Group=myapp

Create the dedicated user account:

sudo useradd --system --no-create-home --shell /bin/false myapp

This creates a system user that can’t log in and doesn’t have a home directory.

Filesystem Restrictions

You can limit what parts of the filesystem a service can access:

[Service]
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/opt/myapp/data
  • PrivateTmp=true gives the service its own private /tmp directory
  • ProtectSystem=strict makes most of the system read-only
  • ReadWritePaths explicitly allows writing to specific directories

Resource Limits

Prevent services from consuming too many system resources:

[Service]
MemoryLimit=512M
CPUQuota=50%

This limits the service to 512MB of RAM and 50% of one CPU core.

Troubleshooting Common Issues

Service Won’t Start

When a service fails to start, the first step is checking its status:

systemctl status myapp

Look for error messages in the output. Common issues include:

  • Configuration file syntax errors
  • Missing dependencies
  • Permission problems
  • Port conflicts

Service Keeps Restarting

If a service keeps restarting, check the logs:

journalctl -u myapp -f

Look for error messages that indicate why the service is crashing. Common causes include:

  • Application bugs
  • Missing configuration files
  • Database connection problems
  • Port already in use

Permission Denied Errors

These often occur when:

  • The service user doesn’t have permission to read configuration files
  • The service user can’t write to log directories
  • SELinux or AppArmor policies are blocking access

Check file permissions and ownership:

ls -la /etc/myapp/
ls -la /var/log/myapp/

Performance and Monitoring

Checking Resource Usage

Monitor how much CPU and memory your services are using:

systemctl status myapp

The status output includes basic resource usage information.

For more detailed monitoring:

systemctl show myapp | grep -E "(CPU|Memory)"

Log Management

Service logs can grow large over time. Configure log rotation to prevent disk space issues:

journalctl --vacuum-time=30d

This removes journal entries older than 30 days.

You can also limit the total size of logs:

journalctl --vacuum-size=1G

Integration with Modern DevOps Practices

Configuration Management

When managing multiple servers, use configuration management tools like Ansible to deploy and manage systemd services consistently:

- name: Deploy application service file
  template:
    src: myapp.service.j2
    dest: /etc/systemd/system/myapp.service
  notify:
    - reload systemd
    - restart myapp

Monitoring and Alerting

Integrate systemd with monitoring systems to get alerts when services fail:

systemctl is-failed myapp

This command returns a non-zero exit code if the service has failed, making it easy to check from monitoring scripts.

Container Integration

While containers are becoming popular, systemd is still useful for managing containerized applications:

[Unit]
Description=My Containerized App
After=docker.service
Requires=docker.service

[Service]
Type=simple
ExecStart=/usr/bin/docker run --name myapp -p 8080:8080 myapp:latest
ExecStop=/usr/bin/docker stop myapp
Restart=always

This manages a Docker container as a systemd service, providing the benefits of both technologies.

Best Practices for DevOps Engineers

Service Design

Keep it simple: Use Type=simple when possible. It’s easier to understand and debug.

Use dedicated users: Never run services as root unless absolutely necessary. Create dedicated system users for each service.

Handle signals properly: Ensure your applications handle SIGTERM gracefully for clean shutdowns.

Implement health checks: Build health check endpoints into your applications so you can verify they’re working correctly.

Configuration Management

Use environment files: Store configuration in separate files that can be managed independently:

[Service]
EnvironmentFile=/etc/myapp/environment

Document dependencies: Clearly document what other services your application needs and configure appropriate dependencies.

Version your unit files: Keep unit files in version control along with your application code.

Monitoring and Maintenance

Set up log monitoring: Configure log aggregation and alerting for error patterns in your service logs.

Monitor resource usage: Set appropriate limits and monitor actual usage to optimize performance.

Plan for failures: Configure appropriate restart policies and test failure scenarios.

Regular maintenance: Periodically review and clean up old services, update configurations, and rotate logs.

Conclusion

Mastering systemd is essential for modern DevOps work. It provides a powerful, unified interface for managing services, logs, and system resources. The key concepts to remember are:

Understanding unit files and how to configure them for your specific needs is fundamental. Service dependencies ensure your applications start in the right order and have the resources they need. Proper logging configuration and log analysis skills help you troubleshoot issues quickly.

Security should always be a priority - use dedicated users, filesystem restrictions, and resource limits to protect your systems. Regular monitoring and maintenance keep your services running smoothly and help you catch problems before they impact users.

Start with simple configurations and gradually add more advanced features as you become comfortable with the basics. Practice creating custom services for your applications, and always test changes in a safe environment before deploying to production.

Systemd might seem complex at first, but its consistency and power make it an invaluable tool for managing modern Linux infrastructure. The time invested in learning it well will pay dividends throughout your DevOps career.