Learning How to Deploy a AlmaLinux VM With Packer (Part 2)

Troubleshooting and Refining the AlmaLinux VM Build With Packer

Learning How to Deploy a AlmaLinux VM With Packer (Part 2)

Recap of Part 1

In Part 1, I started building an AlmaLinux VM using Packer but encountered several challenges during the installation process, including insufficient disk size and memory allocation, as well as issues retrieving the Kickstart file due to network configuration. In Part 1, I outlined the troubleshooting steps taken to address these issues, such as increasing the disk size to 256GB, upgrading the memory to 8GB, and switching from a NAT to a bridged network for improved access to the Kickstart file. While these adjustments allowed the Kickstart to begin executing, the installation was not yet complete, highlighting the ongoing challenges and lessons learned in this iterative process.

Introduction to Part 2

Continuing from the progress made in Part 1, this section dives deeper into the troubleshooting phase of my AlmaLinux VM build using Packer. After successfully addressing some initial configuration challenges, I focused on refining the installation process to ensure the Kickstart file executed smoothly. Despite my efforts, I encountered further hurdles that required critical thinking and problem-solving. In this part, I will detail the specific issues faced, the solutions implemented, and the lessons learned as I work towards successfully completing the VM setup.

Building the AlmaLinux VM

After adjusting the initial configurations, I was ready to build the AlmaLinux VM. It was gratifying to see the installer progress through the early configuration stages and start to actually build the OS. I encountered some initial errors during this phase:

1. First Error: During the installation process, I encountered an error message indicating that there was an invalid auto part type specified in my configuration file. The error was clearly communicated as follows:

invalid auto part type xfs

This message signaled that the system did not recognize the specified auto part type. To resolve this issue, I needed to modify the configuration to align with the expected parameters. After some research, I determined that the default auto partitioning method was set to use XFS, so I updated my configuration line to the following:

autopart # as xfs is the default

2. Error on Line 4: As I continued to refine my configuration, I noted that an installation command I had included was no longer necessary. This was further confirmed by the error message that explicitly stated:

install has been removed

This message indicated that the particular command I was trying to use was deprecated and had been eliminated from the installation process. In light of this information, I resolved the issue by simply removing the problematic line entirely from my configuration file:

install  # Begin installation from the CD-ROM.

3. Installation Source and Software Selection: Initially, the configuration file indicated that the CD-ROM was to be used as the installation source. This was reflected in the original line:

cdrom  # Use the installation media from the CD-ROM.

However, as I’m just using the boot image, it was an error to have the CD-ROM specified at all, as all packages will be downloaded from the repository as the installation progresses. I will get these packages from the same mirror as the disk image, ensuring that the installation process can seamlessly access the necessary resources and updates:

url --url="https://www.mirrorservice.org/sites/repo.almalinux.org/almalinux/9.4/BaseOS/x86_64/os"

Screenshots of the Build Process

After navigating through the configuration hurdles and troubleshooting various issues, the installation completed successfully. The machine rebooted for its initial boot sequence. As I observed the boot process unfold, the AlmaLinux boot screen was replaced by the familiar Linux login prompt. It was gratifying to reach this point after the complexities I had encountered during the setup, and I was not expecting there to be further issues with the closing stages of the build, but there were more issues to come. Monitoring the Packer output, I encountered a timeout error waiting for SSH, which significantly hindered the process. Since Packer relies on SSH to communicate with the VM, the post install actions couldn't complete until this issue was resolved.

Troubleshooting the SSH Issue

Steps Taken to Resolve the SSH Issue:

1. Checked Network Settings: I reviewed the network settings to ensure the VM was properly configured. The network configuration was set to static, with the following parameters:

network --bootproto=static --device=eth0 --ip=192.168.1.187 --netmask=255.255.255.0 --gateway=192.168.56.1 --nameserver=8.8.8.8 --hostname=almalinux-vm

2. Configured SSH for Root Login: I realised that SSH was not allowing password authentication or root logins by default in AlmaLinux, so I added the following lines in the post-installation script:

   %post --interpreter=/bin/bash
   # Configure SSH to allow root login and password authentication
   {
       echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
       echo "PasswordAuthentication yes" >> /etc/ssh/sshd_config
   } >/dev/null 2>&1

   # Restart the SSH service
   systemctl restart sshd >/dev/null 2>&1
   %end

3. Addressed SSH Connection Issues: After making the above changes, I could log in to SSH from the shell with a username and password; however, this did not work from Packer because SSH is hardcoded to use an IP. Changing the ssh_x values had no effect on the setup.
4. Implemented Port Redirection: Packer hardcodes the IP address and tries to connect to the host using a random port that it does not forward to the guest OS. This implementation choice was frustrating. To troubleshoot, I ran Packer in debug mode to identify the port it was trying to use from the logs. I then used netsh from an elevated session to redirect that port:

   netsh interface portproxy add v4tov4 listenport=2714 listenaddress=127.0.0.1 connectport=22 connectaddress=192.168.1.187

5. Resolved Authentication Issues: I found that the hashed password specified in the HCL file was not working, so I temporarily replaced it with a plain text password. This change progressed the SSH connection from failure messages to successfully opening a new SSH session and the build concluded without further errors.

   While successfully deploying a bare-bones AlmaLinux VM using Infrastructure as Code (IaC) principles, I found myself perplexed by the developers' choice to have Packer ignore the specified SSH port, which necessitated manual redirection. As a DBA, encountering such decisions is not unfamiliar to me. Rather than dwell on this puzzling aspect, I shifted my focus to identifying effective solutions to automate the port configuration process, streamlining the workflow and minimisingthe need for manual intervention.

Wrapper Script with Redirection

I developed a PowerShell script that incorporates several key features to facilitate network redirection during a Packer build process. Here’s an overview of its functionality:

6. Environment Setup: The script begins by setting the PACKER_LOG environment variable. This enables detailed logging, which is crucial for tracking the process and identifying any issues, and also the port needed for network redirection.

7. Background Job Management: By using Start-Job, the script allows for non-blocking execution of the Packer build. This enables simultaneous monitoring of the output, which is essential for timely responses to any errors that arise during the build.

8. Dynamic Port Handling: The script intelligently captures and responds to specific error messages generated by the Packer job. By employing a regular expression, it identifies when a port is refused and subsequently sets up a port proxy to redirect traffic to a specified IP address. This dynamic handling improves the robustness of the build process.

9. Using Netsh with Elevated Privileges: Throughout this lab project, I want to apply the principle of least privilege, so I considered using PsExec, which I hoped would allow elevation without triggering UAC. However, it doesn’t work as intended. Since netsh requires elevated permissions to perform the redirect, the only remaining option is to approve the UAC prompt. I opted not to elevate the entire script to maintain this principle

10. Clean Up: At the conclusion of the build, the script ensures that the system is restored to its original state by removing the port proxy. This cleanup step reflects good scripting practices, helping to maintain a tidy and well-managed environment.

At the end of the Packer process, I had an OVF file and the virtual disk images of the VM. The OVF (Open Virtualization Format) file is an XML document that describes the virtual machine's configuration, detailing the virtual hardware, resource requirements, and network settings. This open standard format is designed to facilitate the packaging and distribution of virtual appliances across various platforms. The accompanying virtual disk images contained the actual data and operating system required for the VM to function. Together, these files enable you to deploy the virtual appliance in various environments, ensuring consistency and compatibility. Once you have the OVF file and the disk images, here are several actions you can take to maximise their utility:

1. Testing:

• Spin up the virtual machine in a test environment after importing the OVF file to ensure that everything functions as expected and configurations are correct.

2. Convert to BOX Format for Vagrant:

• Convert the OVF file into a BOX format for Vagrant, allowing for easy sharing and deployment of consistent environments.

3. Create OVA Packages:

• Package the OVF file along with any associated virtual disk files into a single OVA (Open Virtual Appliance) file for simplified distribution.

4. Import into Virtualisation Platforms:

• Import the OVF file into various virtualisation platforms like VMware, VirtualBox, and Microsoft Hyper-V to quickly create a virtual machine.

5. Cloud Deployment:

• Use cloud providers that support the direct import of OVF files for rapid deployment of your appliance in cloud environments. Other options include customising virtual machine settings after import, integrating with automation tools for streamlined processes, and using the OVF file as part of your backup strategy for quick recovery if needed. The flexibility and open standard nature of OVF, being fundamentally an XML file, make it a valuable asset in managing virtual appliances effectively.

Final tests

In this test lab project i will be using Vagrant to automate the deployment of the VMs, however before i do that i wanted to spin up the appliance and make sure everything was working as expected. I imported the OVF into VirtualBox and started the machine. I was greeted with the login prompt and proceded to log in as the damean user, this resulted in being asked to change the password as expected which was excellent news. I then checked the IP address and attempted to ssh to the machine as root

which failed as expeced and i was able to log in as the damean account from an ssh session.

Success

When i started this process I was insptired by Captain Picard, using AI to help me build the template and i used an iterative procsess to learn about packer as i went about building this bare bones Linux Vm and seeing what Packer could accomplish. there were many more obstacles in the way than i was expecting and i am gratified by the results, at the end of this process this has definately been a worthwhile activity and I am inspired by Captain Archer. “It’s been a long road, getting from there to here”

Next Steps

Now that I’ve successfully created the bare-bones AlmaLinux VM, the next challenge will be to build out specific machines for my lab environment using this template as the baseline. The key tasks ahead include:

1. Building a RADV Router for the IPv6-only environment, which will enable stateless address autoconfiguration (SLAAC) for the internal lab machines.
2. Setting up a DNS64 Translation Machine to allow IPv6-only hosts to communicate with IPv4 services using NAT64.
3. Deploying Linux Database Servers that will operate seamlessly in this IPv6 environment, providing a robust and scalable backend infrastructure.

To ensure the smooth operation of this lab environment, careful network configuration will be essential. I will need to modify the baseline NICs and IP settings for each VM to suit the lab's requirements. Additionally, packaging the OVF files with Vagrant will streamline future deployments and facilitate rapid provisioning of these machines.

 All the files refered to in this post