DevOps: Automation worth more than a million!
In this Writer’s Room blog, Andela Community member Shreeharsha G N discusses his experience in DevOps and explains how he developed a Jenkins framework to streamline processes and improve his client’s automation!
Yes, the title may be a small exaggeration: but the technology we’re about to discuss is no misnomer!
As part of our enterprise Solution R&D division, I was assigned a task to automate the “execution of a workload” on different SUTs (System Under Test) with four metrics, which takes nearly 2-3 days to complete execution. Seven of my teammates were scheduling these workloads on 15+ different SUTs almost every week, manually administering each metric one after another, and in-between weekends they would encounter more delays in getting the necessary results.
This exercise continued almost every three months, three times a year, costing the client a lot of money purchasing servers (for their on-prem/colo/hybrid cloud environments).
So, I developed a Jenkins framework that executes these workload metrics scheduled on different SUTs, as and when requests from team members hit a queue, which is freed up every 5 minutes. It was just a ‘wrapper’ over an already automated python package, which was not supporting multi-tenant use cases. This was challenging, as it was a multi-tenant use-case with reports generated for each execution run which had to be tagged to end users/SUTs/metrics and all together, with debugging logs required for each pipeline stage.
Objective and Goal of the Automation:
Concept workflow
Dev DataOps
Automated performance tests scheduling on different SUTs
SUTs –> System Under Test
The flow chart I designed for this use case is below:
Automation Flow-Chart
Applications:
1. Even a not-so-experienced person can execute these applications and therefore, make cost savings.
2. Sanity tests – ILO, SUT, OS, Network, etc.
3. Regression tests: Early performance bugs detection across stacks.
4. Optimization pipelines – RFI, RFP.
5. In-house pipelines tools – Python/Java plugins and more…
Impact:
- Helped to successfully complete deals of RFP requests worth 5 to 8+ million US dollars. (Recent Example: SSA customer bid of 173 million wins in 2 months, an 8-year contract).
- Fast turnaround time for RFP requests and customer delight.
Although I am a linux geek, I had to learn Windows batch scripting, as the automation framework used for the execution of the application, collect logs at the system level and dump the results on an NTFS shared across the team. The initial part was to get users input defined as a key-value pair that would have SUT (system under test) IP, credentials, BIOS tunes, OS tunes, etc., so the Jenkins framework shown above in the flow chart would read these key-value pairs and apply the BIOS tunes, OS tunes to the SUT, followed by application execution, with the choice of the application/metric made in the UI style drop-down in the Jenkins framework itself.
The users would edit and copy their run parameters as a text file and copy to the Input folder, and Jenkins framework would read these files every 5 mins and schedules the application execution on different SUTs automatically. Once the runs are complete, the results are dumped in user-specific folders with tuning-comment strings to identify users who executed a particular application on a given SUT, with specific tuning applied. This history of tunes and application results can be used to build machine learning models and use them for recommendations to RFP/RFI requests. However, if there are errors in the execution, it is currently not handled but is notified through Slack from the Jenkins workflow.
Some screenshots of the automation are shown below:
Input folder with test*.dat file containing key-value pairs
Results of various systems and applications like intspeed,fpspeed fprate etc with jadmin user.
The pipeline can also be launched manually with a choice of different systems.
Jenkins pipeline used for above
Since the SUTs on which the application is launched had to be on an isolated network (for performance tests), the JVM parameters used to launch are as below, so that there is no impact of Jenkins agents on the performance results.
Matrix based users authentication
Different app choices
Slack notification plugin in Jenkins to report test workflow results.
The rest of the Python, windows batch scripts and Jenkins groovy scripts can be found at this GitHub link.
Want to be part of a vibrant tech community? Then join the Andela Talent Network!
Find Work
Related posts
The latest articles from Andela.
.png)
Accelerate your ambition: 7 ways to spark career growth
The 6 most in-demand tech skills are also the hardest to find
Andela Research Finds Increasing Demand for Global Remote Tech Talent
We have a 96%+
talent match success rate.
The Andela Talent Operating Platform provides transparency to talent profiles and assessment before hiring. AI-driven algorithms match the right talent for the job.
