Instances using SSH Keys are configured to run OpenSSH server leaving them exposed to OpenSSH attacks, lacking the features of other access methods and reinforcing the use of pet style infrastructure.
SSH is a common tool for accessing remote servers, but it has been specifically targeted for attacks and has known vulnerabilities. While firewall rules and placing SSH targets behind a load Balancer can mitigate these risks for targeted and time-sensitive connections, AWS offers an alternative tool for secure terminal access via AWS APIs: the AWS SSM Agent. The AWS SSM Agent provides several advantages over direct SSH access, such as the following:
Additional Terminal Functionality
In addition to providing terminal access, AWS SSM Agent also records commands and allows for logging and interrogation of commands run on the shell session. This provides an audit trail of actions performed on the instance and allows for detection tools to fetch all SSM logs from AWS Cloudwatch rather than shipping all SSH Server logs from all machines to a central point.
Moreover, AWS SSM Agent supports templates for commands that can be populated and distributed to multiple machines at once, similar to Ansible usage across multiple hosts. This functionality enables efficient mass management and automation of multiple instances.
Centralised access controls via AWS IAM
Access to machines is controlled by AWS IAM permissions, allowing shell access to be integrated into a single point of authentication and authorisation. If your AWS IAM is tied to single sign-on (SSO) or external identity services, there is no need to separately manage SSH keys as AWS IAM roles can provide authorisation to initiate terminal sessions over AWS SSM.
This centralised approach simplifies access management, enhances security by reducing the risk of key mismanagement, and ensures consistent access control policies across your infrastructure.
Enhanced Security
Accessing an AWS SSM-induced terminal on instances is substantially more difficult and has fewer available tools compared to attacking SSH server applications. Utilising AWS SSM introduces multiple layers of protection, making it a more secure alternative to direct SSH access to the SSH server.
Private Subnet Usage
OpenSSH Server requires network connectivity from the client to the server to operate. In contrast, AWS SSM only requires connectivity between the instance and the AWS SSM service endpoint, which is hosted within AWS data centers. This means that AWS SSM clients can effectively run on instances inside private subnets with access to the SSM endpoint service available via AWS VPC endpoints.
While OpenSSH Server has been secure for decades, its prominence and ubiquity make it a frequent target for attacks. There have been recent instances where attackers have successfully implemented malware directly into OpenSSH Server software which has been distributed to users.
If OpenSSH is compromised, servers running the OpenSSH Server can become immediately vulnerable.
AWS SSM runs the same risk, however, as the code base for OpenSSH has already been compromised by malicious open source contributions it is worth considering this risk as well as the motivations of the malicious actors. Successful compromise of OpenSSH Server offers a substantial user base to plunder.
SkySiege’s AWS Vulnerability Scan automatically detects this vulnerability across all AWS Regions with the report delivered the same day.
Whilst removal of OpenSSH is simple, the release and management of accessible shell access is a more difficult and particular manner depending on the governance and management of the organisation. In simple terms - SSH keys are known and understood by most developers, however replacement tools are unlikely to be as well known and will almost always incur some level of learning and adoption. Additionally, the distribution and management of credentials is also important, often requiring a centralised strategy and process implementation for larger teams.
For any current instances you can access the instance and stop the OpenSSH Server depending on the service controlling the OpenSSH process. Usually this would be systemd process such as ssh therefore requiring the below commands.
Note that once the SSH process is disabled any new connections to the SSH client won't be possible.
Once the current connection is terminated the instance will be unreachable via SSH
Prior to shutting off OpenSSH access, it’s important to either provision another process to enable connectivity and control of the instance or to have the full operating processes enabled to cease direct control of the instance and to treat the instance as a cattlised resource.
It is possible to lock out the instance by disabling OpenSSH, requiring a more convoluted restart and reprovision via CloudInit to rescue the server. Rather than attempt to immediate fix this issue it’s better to:
Implementing a new process and governance would likely result in replacement instances, ideally in a cattle-managed format. However, if you’re still running instances that need in-place updates then you should continue with commands such as the below to stop and disable your OpenSSH server.
# Stopping the SSH Server
$ systemctl stop ssh
# Disabling the SSH Server from restarting as part of automated restarts
$ systemctl disable ssh
Installation of AWS SSM takes different forms depending on your server operating system, package management and appetite for supply chain management. For larger scale enterprises it benefits creating a full image creation chain to allow for prebaked machine images to be distributed to accounts. For smaller enterprises it’s simple to include the installation commands as part of image user data. This potentially replicates your configuration commands across multiple locations and could cause maintenance issues in the future, but for a smaller estate this can be preferable to acquiring the skills and pipelines to control a full end to end system build.
AWS provides documentation on the installation processes for all supported operating systems. However, this only covers the technical installation of the tool and not the dissemination of governance, documentation and tooling to make the most of AWS SSM.
For example, there are templated scripts, port forwarding, parameter storage and other features that allow for enhanced centralised control and management of your server estate. Depending on your appetite, you may make these scripts available across your accounts for management of all machines. This can include commands such as automated patches, pruning logs, or restarting AWS Cloudwatch Logs Agent.
As always, organisation change needs appropriate guidance and support to ensure successful adoption. Getting an experienced architect onboard to guide your migration is a step to success: