Simson Chow

In this article, I’ll share how to create a custom Striim CentOS image with Packer, deploy it, and incorporate it into your infrastructure and DevOps stack. Before we set up our environment and start building and deploying the image, let’s go through the definitions of both tools.

What is HashiCorp Packer?

Hashicorp Packer is an open-source Infrastructure-as-Code (IaC) tool that enables you to quickly build and deploy custom images for cloud and on-premises environments. With Packer, you can create custom image builds for a variety of platforms, such as Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, and more. You can use Packer to automate the process of building images from scratch, including creating, configuring, and optimizing them. Additionally, Packer can be used to efficiently deploy those images in multiple cloud locations or on-premises.

Packer is great for automating the creation of machine images so that they can be deployed quickly and easily. The tool supports a variety of image formats and has built-in support for various configuration management tools, such as Chef and Puppet. By using Packer, you can ensure that your images are always up-to-date and properly configured. This makes it easier to manage and maintain your infrastructure in the cloud or on-premises.

What is Striim?

Striim is an end-to-end streaming platform for real-time data integration, complex event processing, and analytics. It is designed to ingest, analyze, and deliver massive volumes of data from multiple sources including databases, files, messaging systems, and IOT devices. 

By integrating Packer into the Striim deployment process, we can quickly create an automated environment that enables your analytics team to immediately replicate real-time data to data warehouses and/or RDBMS databases.

Pre-requisites:

1. An available Linux (CentOS, Ubuntu, or Suse) machine.
2. Install Packer in the machine. More information: https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli
3. Create a Service Account in GCP with Compute Instance Admin (v1) and Service Account User roles attached to it and generate the key. More info: https://blog.knoldus.com/how-to-create-a-custom-image-using-packer-in-gcp/#create-a-service-account-in-gcp
4. A Striim license. 

 

Setting Up Your Environment

Log in to your VM and verify Packer is installed by running this command:

$ packer --version
1.8.6
Once verified, copy the JSON key from the GCP service account to the home directory:

$ ls
account_key.json

Export the following environment variables for later configuration use:

# Values from the Striim license
export company_name=<company_name_from_striim_license>
export cluster_name=<cluster_name_from_striim_license>
export license_key=<license_key_from_striim_license>
export product_key=<product_key_from_striim_license
>

# Setting up the passwords for Keystore, admin, and sys users
export keystore_pass=<keystore_pass_for_striim_config>
export admin_pass=<admin_pass_for_striim_config>
export sys_pass=<sys_pass_for_striim_config>
export mdr_type=<mdr_type_for_striim_config>

 

Creating Your Striim Image

Create a shell script named striim_install.sh and add the following code to it:

 

Striim (V4.1.2) and Java JDK (V1.8.0) will be installed using this shell script, and Striim will be configured using the environment variables that we specified in the previous section. Note: This shell script installs Striim and Java JDK only in CentOS, RedHat, Amazon Linux 2, and SUSE Linux machines.

Create a JSON file named packer_striim_image.json and copy the following code:

The variables section declares variables for Packer from the environment variables that we exported in the previous section. More info: https://developer.hashicorp.com/packer/guides/hcl/variables

The builders section allows us to define the cloud provider and provide information about our GCP project. More info: https://developer.hashicorp.com/packer/plugins/builders/googlecompute

The provisioners section uses built-in and third-party software to install and configure the image after booting. In our case, we are copying our striim_install.sh to the machine’s /tmp/ directory and executing the script to install Striim and its dependencies. More info: https://developer.hashicorp.com/packer/docs/provisioners

Once these files are created, we should see a file structure like the one below in our home directory:

----- home/
-- gcp_account_key.json
-- packer_striim_image.json
-- striim_install.sh

Deploying Your Image to Google Cloud Platform

Let’s run the following commands to validate our Packer JSON file:

$ packer validate packer_striim_image.json
The configuration is valid.

Once we verified that it is valid, we can deploy our image:

$ packer build packer_striim_image.json

...
==> Wait completed after 3 minutes 36 seconds

==> Builds finished. The artifacts of successful builds are:
--> googlecompute: A disk image was created: gcp-custom-striim-image-167888819

We can copy the image name gcp-custom-striim-image-167888819 and use it to build a VM in GCP to check that Striim is correctly installed on this image: 

We can access the Striim UI by navigating to <public_or_private_ip>:9080 in the browser once the VM is in the “Running” state:

In conclusion, Packer is a powerful tool that can be used to create a more efficient infrastructure-as-code approach. With Packer, image builds and deployments can be automated in a secure and consistent manner. This automation allows you to quickly build and deploy Striim images without worrying about manual configuration errors. In addition to this, using Striim Cloud can fully automate this entire process. Visit the Striim Cloud page for a fully managed SaaS service/solution and Pay As You Go option to reduce total cost of ownership.

Deploying a server can be a time-consuming process, but with the help of Terraform, it’s easier than ever. Terraform is an open-source tool that automates the deployment and management of infrastructure, making it an ideal choice for quickly and efficiently setting up a Striim server in the cloud or on-premise. With the help of Striim‘s streaming Extract, Transform, and Load (ETL) data platform, data can be replicated and transformed in real-time, with zero downtime, from a source database to one or more target database systems. Striim enables your analytics team to work more efficiently and migrate critical database systems. 

In this blog post, we’ll walk through the steps of how to use Terraform to automate the deployment of a Striim server in AWS.

Pre-requisites

1. Access to an AWS account including the Access Key ID and Secret Access Key. 
2. Have an available Linux machine.
3. General understanding of what Striim is.
4. A Striim license. For free trials, go to https://signup-developer.striim.com/.

Install and Setup Terraform

In order to automate the deployment of a Striim server, we’ll first need to install Terraform on our CentOS Linux machine. 

Let’s log in to it and enter the following commands into the terminal:

sudo yum install -y yum-utils
sudo yum-config-manager - add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

If you’re using a different operating system, please find the appropriate instructions in this link: https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli

We’ll be using Terraform version 1.3.6 in this tutorial. Please verify the version by running this command:

terraform -version

Terraform v1.3.6 on linux_amd64If

Once the installation is successful, we can authenticate to our AWS account by exporting the AWS_ACCESS_KEY_ID , AWS_SECRET_ACCESS_KEY and AWS_REGION environment variables:

export AWS_ACCESS_KEY_ID=123456789-1234-1234-222
export AWS_SECRET_ACCESS_KEY=123456789-234-234-444
export AWS_REGION=us-west-2

For more information about getting your AWS access keys from an IAM user, please visit this link: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

Configure Terraform

After the installation process, we can create a directory named striim_server_tf and add the following files inside:

• main.tf — will include the primary set of configuration for your module. Additionally, you can create additional configuration files and arrange them in a way that makes sense for your project:

• variables.tf — will contain the variable definitions for your module:

As was mentioned above in the “Striim Credentials and License Information” section from the variables.tf file, we will need to set Striim’s license information and user passwords as environment variables since they are confidential values:


export TF_VAR_striim_product_key=123456-123456-123456
export TF_VAR_striim_license_key=123456-123456-123456-123456-123456-123456-123456-04C
export TF_VAR_striim_company_name=striim
export TF_VAR_striim_cluster_name=striim_cluster_name
export TF_VAR_striim_sys_password=my_awesome_password
export TF_VAR_striim_keystore_password=my_awesome_password
export TF_VAR_striim_admin_password=my_awesome_password
export TF_VAR_striim_mdr_database_type=Derby

Terraform will then be instructed to search for the variable’s value in the environment variable by the TF_VAR_ prefix. More information: https://developer.hashicorp.com/terraform/cli/config/environment-variables

Once we have these files created, we should see a directory and file structure like this:

striim_server_tf
|
|-- main.tf
|
|-- variables.tf

Run Terraform

At this point, we have configured our Terraform environment to deploy a Striim server to our AWS account and written Terraform code to define the server. To deploy it, we can now execute the two Terraform commands, terraform plan and terraform apply, inside of the striim_server_tf directory.

• The terraform plan command lets the user preview the changes (create, destroy, and modify) that Terraform plans to make to your overall infrastructure.
• The terraform apply command executes the actions proposed in a Terraform plan. 

If these commands executions are successful, you should see a message at the end of your terminal with the following message:


Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Verify the Deployment

To verify the Striim server deployment, navigate to the AWS EC2 console and search for striim-server:

Make sure it’s in a Running state and Status check is 2/2 checks passed.

Next, enter the public IP address of the server with :9080 at the end of the url in a web browser and check to see if Striim is up and running:

Enter your credentials and verify you can log in to Striim console:

By leveraging Terraform and its Infrastructure-as-Code approach, deploying a Striim server can be automated with ease. It allows organizations to save time and money by quickly spinning up Striim servers, which can be used for data migration or zero downtime replication. This blog post provided an overview of how to use Terraform to set up and deploy a Striim server, as well as how to verify that the deployment was successful. With Terraform, it is possible to automate the entire process, making it easier than ever to deploy and manage cloud infrastructure. In addition to this, using Striim Cloud can fully automate this entire process. Visit the Striim Cloud page for a fully managed SaaS service/solution and Pay As You Go option to reduce total cost of ownership.