Mastering systemd: Running Custom Apps as Linux Services

reddit [#143] Created with Sketch.
Venkat Vaddineni
Venkat Vaddineni
1 min read

Mastering systemd: Running Custom Apps as Linux Services

Learn how to create and manage .service files to deploy apps like a pro, automate restarts, and integrate your services into the Linux ecosystem — the DevOps way.

Introduction: Why systemd Matters

Imagine you’ve built an amazing web application or background service. It works perfectly when you run it manually, but what happens when your server restarts? Your app stops, and you have to remember to start it again manually. That’s where systemd comes to the rescue.

systemd is the service manager that powers most modern Linux distributions. It’s like having a personal assistant for your applications that can start them automatically, restart them if they crash, manage their logs, and much more. By the end of this guide, you’ll transform from manually running apps to managing them like a seasoned DevOps professional.

What is systemd and Why Should You Care?

systemd (system daemon) is the init system and service manager for Linux. Think of it as the conductor of an orchestra, coordinating all the services that make your Linux system work. It replaced older init systems and brought many improvements:

  • Automatic startup: Your apps start when the system boots
  • Crash recovery: Failed services restart automatically
  • Dependency management: Services start in the right order
  • Resource control: Limit CPU, memory, and other resources
  • Logging integration: Centralized log management with journald
  • Security features: Run services with minimal privileges

The beauty of systemd is that once you create a service file, your application becomes a first-class citizen in the Linux ecosystem, managed just like built-in system services.

Understanding Service Files: The Blueprint

A systemd service file is a configuration file that tells systemd how to manage your application. These files live in /etc/systemd/system/ and have a .service extension. Think of them as recipes that describe:

  • What your application is and where to find it
  • How to start and stop it
  • What to do if it crashes
  • When it should start relative to other services

Every service file follows a specific structure with three main sections:

[Unit] - Describes the service and its relationships [Service] - Defines how to run the service [Install] - Specifies when and how to enable the service

Creating Your First Service File

Let’s start with a practical example. Suppose you have a simple Python web app that you want to run as a service.

Step 1: Prepare Your Application

First, let’s create a simple Python web server to work with:

# /opt/myapp/app.py
from http.server import HTTPServer, SimpleHTTPRequestHandler
import os

class MyHandler(SimpleHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()
        self.wfile.write(b'<h1>Hello from My Service!</h1>')

if __name__ == '__main__':
    port = int(os.environ.get('PORT', 8080))
    server = HTTPServer(('0.0.0.0', port), MyHandler)
    print(f'Server running on port {port}')
    server.serve_forever()

Step 2: Create the Service File

Now, let’s create a service file for this application:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Custom Web Application
After=network.target
Wants=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment=PORT=8080
ExecStart=/usr/bin/python3 /opt/myapp/app.py
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Step 3: Set Up the Environment

Before we can use the service, we need to set up the proper environment:

# Create a dedicated user for security
sudo useradd --system --no-create-home --shell /bin/false myapp

# Create the application directory
sudo mkdir -p /opt/myapp
sudo chown myapp:myapp /opt/myapp

# Copy your application files
sudo cp app.py /opt/myapp/
sudo chown myapp:myapp /opt/myapp/app.py

Breaking Down the Service File

Let’s understand each part of our service file:

[Unit] Section

  • Description: A human-readable description of your service
  • After: Ensures your service starts after the network is available
  • Wants: Indicates a weaker dependency (nice to have, but not required)

[Service] Section

  • Type=simple: The most common type where the main process doesn’t fork
  • User/Group: Run the service as a specific user for security
  • WorkingDirectory: Set the current directory for your application
  • Environment: Set environment variables
  • ExecStart: The command to start your service
  • ExecReload: Command to reload the service (optional)
  • Restart=always: Restart the service if it crashes
  • RestartSec: Wait 3 seconds before restarting
  • StandardOutput/StandardError: Send logs to systemd’s journal

[Install] Section

  • WantedBy: Determines when the service should start (multi-user.target means normal system startup)

Managing Your Service

Once your service file is created, you can manage it with these commands:

Enable and Start the Service

# Reload systemd to recognize the new service
sudo systemctl daemon-reload

# Enable the service to start at boot
sudo systemctl enable myapp.service

# Start the service now
sudo systemctl start myapp.service

Check Service Status

# View detailed status
sudo systemctl status myapp.service

# Check if the service is running
sudo systemctl is-active myapp.service

# Check if the service is enabled
sudo systemctl is-enabled myapp.service

Control the Service

# Stop the service
sudo systemctl stop myapp.service

# Restart the service
sudo systemctl restart myapp.service

# Reload the service (if ExecReload is defined)
sudo systemctl reload myapp.service

# Disable the service from starting at boot
sudo systemctl disable myapp.service

Advanced Service Configuration

Service Types

systemd supports different service types depending on how your application behaves:

  • simple: Default type, process doesn’t fork
  • forking: Process forks and the parent exits
  • oneshot: Process exits after completing its task
  • notify: Process signals when it’s ready
  • idle: Waits for other services to finish

Resource Limits

You can control resource usage to prevent runaway processes:

[Service]
# Limit memory usage to 512MB
MemoryLimit=512M

# Limit CPU usage to 50%
CPUQuota=50%

# Limit number of file descriptors
LimitNOFILE=1024

# Limit number of processes
LimitNPROC=10

Environment Variables

Multiple ways to set environment variables:

[Service]
# Single variable
Environment=PORT=8080

# Multiple variables
Environment=PORT=8080 DEBUG=true

# Load from file
EnvironmentFile=/etc/myapp/config

# Combine both
Environment=PORT=8080
EnvironmentFile=/etc/myapp/config

Security Hardening

systemd provides many security features:

[Service]
# Run with minimal privileges
NoNewPrivileges=true

# Restrict access to /home, /root, and /run/user
ProtectHome=true

# Make /usr, /boot, /etc read-only
ProtectSystem=strict

# Create private /tmp
PrivateTmp=true

# Restrict network access
RestrictAddressFamilies=AF_INET AF_INET6

# Prevent access to kernel modules
ProtectKernelModules=true

Real-World Examples

Node.js Application Service

[Unit]
Description=Node.js Web App
After=network.target

[Service]
Type=simple
User=nodeapp
WorkingDirectory=/opt/nodeapp
Environment=NODE_ENV=production
Environment=PORT=3000
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Docker Container Service

[Unit]
Description=My Docker Container
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/docker run -d --name mycontainer -p 8080:8080 myimage:latest
ExecStop=/usr/bin/docker stop mycontainer
ExecStopPost=/usr/bin/docker rm mycontainer

[Install]
WantedBy=multi-user.target

Background Worker Service

[Unit]
Description=Background Job Processor
After=network.target

[Service]
Type=simple
User=worker
WorkingDirectory=/opt/worker
ExecStart=/opt/worker/process_jobs.py
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

# Resource limits for background workers
MemoryLimit=256M
CPUQuota=25%

[Install]
WantedBy=multi-user.target

Troubleshooting Common Issues

Service Won’t Start

Check the service status first:

sudo systemctl status myapp.service

Common issues and solutions:

  1. Permission errors: Ensure the service user has access to files and directories
  2. Missing dependencies: Check if required packages or services are installed
  3. Wrong paths: Verify ExecStart path and WorkingDirectory exist
  4. Syntax errors: Use systemctl daemon-reload after editing service files

Service Keeps Restarting

# View recent logs
sudo journalctl -u myapp.service -n 50

# Follow logs in real-time
sudo journalctl -u myapp.service -f

Look for error messages in the logs to identify the root cause.

Service Doesn’t Start at Boot

# Check if service is enabled
sudo systemctl is-enabled myapp.service

# Enable if not already enabled
sudo systemctl enable myapp.service

# Check what target it's installed to
systemctl list-dependencies multi-user.target | grep myapp

Monitoring and Logging

Using journalctl

systemd integrates with journald for centralized logging:

# View all logs for your service
sudo journalctl -u myapp.service

# View logs from today
sudo journalctl -u myapp.service --since today

# View logs from the last hour
sudo journalctl -u myapp.service --since "1 hour ago"

# Follow logs in real-time
sudo journalctl -u myapp.service -f

# View logs with priority level
sudo journalctl -u myapp.service -p err

Log Rotation

journald handles log rotation automatically, but you can configure it:

# Check journal disk usage
sudo journalctl --disk-usage

# Configure maximum journal size in /etc/systemd/journald.conf
SystemMaxUse=100M

Best Practices

Security First

  • Always run services with dedicated, non-privileged users
  • Use systemd’s security features like ProtectHome and ProtectSystem
  • Set resource limits to prevent resource exhaustion
  • Store sensitive configuration in files with restricted permissions

Reliability

  • Use Restart=always for critical services
  • Set appropriate RestartSec values (not too aggressive)
  • Implement proper health checks in your application
  • Use Type=notify if your application supports it

Maintainability

  • Write clear descriptions and comments in service files
  • Use consistent naming conventions
  • Keep service files in version control
  • Document any special configuration requirements

Monitoring

  • Send logs to journald for centralized management
  • Monitor service status with external tools
  • Set up alerts for service failures
  • Regularly review logs for issues

Conclusion

You’ve just learned how to transform any application into a professional, production-ready Linux service. By mastering systemd service files, you’ve gained the ability to:

  • Automatically start applications at boot time
  • Recover from crashes without manual intervention
  • Manage resources and security effectively
  • Integrate with Linux logging and monitoring systems
  • Deploy applications like a seasoned DevOps professional

The journey from manually running applications to managing them as services represents a significant leap in operational maturity. Your applications are now resilient, manageable, and ready for production environments.

Start small with a simple application, then gradually incorporate advanced features like resource limits and security hardening. Before you know it, you’ll be managing complex service ecosystems with confidence.

Remember, every professional service you interact with daily - web servers, databases, monitoring tools - all run as systemd services. You now have the knowledge to join their ranks and manage applications the way they were meant to be run in production Linux environments.

Welcome to the world of professional Linux service management!