1 Take-Blip, Belo-Horizonte, Brazil.
2 Independent Researcher, Finland.
Review Article
World Journal of Advanced Research and Reviews, 2023, 19(03), 1639-1649
Received on 25 July 2023; revised on 09 September 2023; accepted on 12 September 2023
This study explores the optimization of data engineering workflows through the integration of Continuous Integration and Continuous Deployment (CI/CD) practices, utilizing Terraform to reduce deployment time and operational overhead. The purpose of the research is to examine how Terraform's Infrastructure as Code (IaC) capabilities can streamline infrastructure provisioning and enhance the efficiency of data engineering processes. The study employed a comprehensive review of current CI/CD practices, analyzing the impact of Terraform on deployment efficiency, operational complexity, and regulatory compliance. Through an in-depth analysis, key findings reveal that Terraform significantly enhances automation, resulting in reduced manual intervention, faster deployment cycles, and consistent infrastructure management. However, the study also identifies challenges, including regulatory compliance, data security, and the need for skilled personnel, as barriers to the effective implementation of CI/CD optimization. To address these challenges, the study recommends a strategic approach that includes integrating AI and machine learning into CI/CD pipelines, enhancing compliance and security measures, and fostering a cultural shift within organizations to adopt agile development practices.
The study concludes that the optimization of CI/CD processes using Terraform offers a robust framework for improving the agility and security of data engineering workflows. Nevertheless, it emphasizes the need for continuous research and innovation to address regulatory, operational, and inclusivity challenges. The study recommends a proactive approach to compliance, ongoing workforce training, and the exploration of CI/CD integration in diverse global settings. By advancing these strategies, organizations can achieve a more resilient and efficient data engineering environment, aligning with the evolving demands of the digital landscape.
CI/CD; Data Engineering; Terraform; Infrastructure as Code; Deployment Efficiency; Operational Overhead
Preview Article PDF
David Akokodaripon and Godwin Nzeako. Enhancing data engineering workflows with CI/CD process optimization using terraform: Reducing deployment time and operational overhead. World Journal of Advanced Research and Reviews, 2023, 19(3), 1639-1649. Article DOI: https://doi.org/10.30574/wjarr.2023.19.3.1795