Patching in AWS (Linux Edition)

While I always encourage readers to leverage AWS managed services or serverless solutions, there are situations where self-managed instances become unavoidable. The reasons for this can vary — some better than others — but the challenges of self-managed environments go beyond just ensuring availability. If you’ve ever been overwhelmed by AWS Inspector’s relentless stream of vulnerability alerts, you’re not alone. In this post, I’ll share two effective ways to patch a typical Linux instance. These methods have been tested on Amazon Linux 2 and Amazon Linux 2023.

The manual but trusty way

SSH/SSM into the instance, run the following commands one by one, each line is a new command.

sudo su

yum update -y

Check the kernel version:

uname -r

List all the kernel version currently installed:

rpm -qa | grep kernel

Remove kernels until only the current kernel and the kernel to install are left. This is to prevent confusion in the system on what to update to.

yum remove <kernel version from above command> -y

Reboot and tada..you patched the instance fully! Just wait a bit and inspector should clear your instance.

The automated but “not-so-sure” way

AWS has been touting their Systems Manager — Patch Manager for years. But seriously, that SH*T is as stable as their SSM can be, which is unstable as SH*T. A slightly higher memory or CPU usage during your patching, boom, your instance just lost connection SSM. So although this on paper comes in handy for managing patching over large number of instance, in reality, I don’t have the best experience with it. However, for completeness sake, I will go over it, but I just want you to be aware of the caveats.

Side note: I assume that you are using SSM, with the relevant permissions and networking done up. If you are not sure what SSM is, then you are (god bless you) not the target audience here. This part is more for my 😢 enterprise gang.

Create baseline

A baseline is basically just what level you want to patch your instance. You can of course use the AWS default baseline but sometimes, you want non security updates too, eg, python libraries. Typically to be safe, I will create my own baseline as follows, leaving all other settings untouched.

side note: as of writing this story on Dec 2024, the patch baseline screen is MF difficult to find on the UI. So if you can’t find it, just type the URL: <region>.console.aws.amazon.com/systems-manager/patch-manager/baselines?region=<region>

Create patch policy — Scheduled Patching

Scheduled patching is like a cron job where AWS Patch Manager will be triggered and start patching your instances. I typically like to select “reboot if needed” because some kernel patches just won’t work without it and may even corrupt the instance without reboots. As for the other settings, I think they are pretty self explanatory so I won’t go over them.

Patch now — On demand Patching

On demand patching is an once off patching job that you want to execute now. The options are pretty self explanatory too.

Bonus — kernel refuse to update

Sometimes the kernel just refuse to update even though you tried everything. Fret not, I’ve been in your shoes too. Here is one of the gazillion things I would try first, because most of the time its the bootloader.

sudo su
mv /boot/grub2/grub.conf /boot/grub2/bk_grub.conf
yum -y update && yum -y reinstall kernel
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot