Services: the thing that restarts your app at 3am

A systemd service file tells systemd what to run, as which user, and what to do when it stops. Restart=always is the line that survives a crash.

In block 02 you killed a hand-started process and nothing brought it back. Here is what would have: a unit file, a short note telling systemd what to run and what to do when it stops. One line in it separates a five second blip from a morning of downtime. Toggle it and find out.

/etc/systemd/system/hello-api.service
Click either highlighted line to change it. That is a real unit file, not a simplified one.
Read Active and Main PID. Tasks, Memory, CPU and CGroup are plumbing, like the Netid and State columns in block 06.
$ journalctl -u hello-api.service

What you just learned

A unit file is not a script and not a program. It is a short description: what to run (ExecStart), who to run it as (User), and what to do when it stops (Restart). systemd reads that description and takes responsibility for the process, which is the part you were doing by hand before.

That is why Restart=always matters so much. Without it, a crash at 3am is downtime until you wake up. With it, systemd notices within a second, starts a fresh copy, and writes down that it did. The PID changing is the proof: it is not the same process that came back, it is a new one wearing the same name.

Two habits come out of this. systemd is not infinitely patient: a service that fails over and over gets stopped with Start request repeated too quickly, which means "your app is broken", not "systemd is broken". And after you edit a unit file, systemd keeps running the old version until daemon-reload, which is most of the "I changed it and nothing happened" moments in self-hosting.

A night shift supervisor. You leave a note saying what the job is and whether to wake somebody if it stops. Then you go to bed, and the note is what decides whether the machine is running when you wake up.
You can name it now: say this to your AI
Write me a systemd unit file for my app that runs as a non-root user, restarts automatically if it crashes, and starts on boot.
Three requirements, one file. You now know why each of the three is in there, which means you can tell when the answer is missing one.

Seen on a real server

hello-api.service: Start request repeated too quickly.
It failed five times in a row and systemd stopped retrying. Your app is broken, not systemd: fix it, then start it again.
Active: failed (Result: signal)
The process was killed and no Restart= line brought it back. Main PID reads 0.
hello-api.service: Scheduled restart job, restart counter is at 1.
Restart=always doing its job: a new process with a new PID, wearing the same name.

systemd has been writing everything down this whole time. Next: how to read it back, and how to find one error in a thousand lines.