Categories
Leadership

Spend (And Enablement) Must Be Intentional

This article was previously published on LinkedIn.

For most companies, there is no room for projects built on hope and wonderment – every dollar matters. Today’s marketplace is unforgiving. Shrinking markets, opportunities, and budgets are forcing difficult decisions. Many companies have already struggled with the choices to furlough, layoff, or otherwise change company spend.

Now is the perfect (and expected) time to analyze spend and understand what investments are intentional, and what investments are designed to bring value to the company. It is also the ideal time to cull projects/spend that seem appealing but have no discernible value in today’s marketplace.

Let’s look at a simple example.

Examine a Typical Digital Transformation Project

Your IT budget is USD10MM. Said budget spans typical OPEX and CAPEX subjects, including people, benefits, hardware, software, office miscellaneous, travel, and training. The number of corporate strategic objectives for the year requires more project capacity than your headcount. Each of the lines of business (LOB) want their needs addressed, each of them vying for priority over the other. Some of us call this a typical Thursday.

A few folks on your team have ideas that may potentially transform your IT operation, enabling quicker response times, more straightforward systems changes, and the flexibility to adapt to the changing winds of the business happening now and in the anticipated future.

It appears the idea may simplify operations and lower long-term spend. While everyone else is fielding daily lights-on operations, handling one-to-many (1:N) projects, participating in meetings, and performing hands-on-keyboard technical things in fits and starts, you green light your lieutenant (Lt.) to explore new ideas looking forward to the future.

Across the next days and weeks, the exploration project updates happen in hallways, at lunch, over a few coffees, and dinner one night. The updates are exciting, new tool possibilities pretty cool, and the potential for a modern architecture changing the operation for the better are enlightening.

As weeks turn into months, the excitement builds as new programming languages, tools and ecosystems coalesce into a working prototype of things promising an upgraded tomorrow. Your Lt. asks for permission to bring a couple of other folks on board the exploration project to increase velocity. You approve. They’ve privately named their project, Rocket Man, to illustrate the value of this project promising to propel them into new territory. Meanwhile, your role as CTO continues to require constant progress on Line Of Business (LOB) projects, executive meetings, prioritization activities, capacity challenges, attrition, and budget revisits due to market shifts.

Nine months elapse. Seven people are now full-time dedicated to the informal, doesn’t formally exist yet project, which has consumed 15 percent of your annual budget, and you need answers. Finance is asking why seven folks are logging time against overhead time codes, LOB leaders are knocking down your door for solutions, and your CEO is hearing some hallway chatter making her curious how you’re doing.

At one point, a peer jokingly jabs asks about your kids and their toy-box project, but you smile, brush it off, and redirect the conversation.

Post-haste, you ask your Lt. to provide a purposeful briefing to understand money spent, time elapsed, people, responsibilities, progress, and definition of done. You privately wonder how this effort made it so far down the road.

The team presents a sharp, polished presentation deck complete with video clips explaining tool chains, sharply drawn architectural diagrams, a few technical cartoons, code snippets in new programming languages not used in today’s operations, and a component architecture status report showing percentages of things done, in progress and not started. They smile with the pride of success, turn off the presentation, turn to you, expect validation, encouragement, and continued support. They feel they have worked hard and performed admirably.

CTO: How do you know when you’ll be done?

Lt.: We’re working to decide on the final programming languages, tools, and solidify the architectural direction. We will have answers in the next couple of weeks.

CTO: When will this be ready for production?

Lt.: We don’t know yet. We’re still working out the plan.

CTO: We’ve spent USD1.5MM to date. This spend is now on the radar of the larger company. How much money do you need to finish the prototype?

Lt.: We’ll come back to you with a number.

At this point, the CTO knows a bunch of money is spent, doesn’t know what he has, when he will have something usable, or when he will be done spending money. At the same time, he is behind on LOB prioritized and funded projects, is still fixing financial spend details with the CFO, and needs to manage the message with his boss, the CEO.

The options appear to be:

  1. Formalize it, publicly communicate and fund it
  2. Kill it, shelf it for later, and tag it as R&D with Finance
  3. Let it go on for a while longer until the answer reveals itself

What did the CTO do well, and what must the CTO change?

What the CTO Did Well

  1. Encouraged innovation
  2. Trusted and supported his team
  3. Gave people latitude to learn, grow, and become more
  4. Invested in the future proactively

What the CTO Must Change

  1. Trust people while additionally setting up engagement parameters
  2. Define desired outcomes in advance of engagement
  3. Set and manage Keep Going / Let’s Stop triggers
  4. Set, manage and communicate expectations laterally and upline

Read Leadership in Absentia for additional suggestions on ensuring people are empowered to grow, learn, and care about successful outcomes within a premeditated framework.

10 Recommendations Enabling Exploration and Discovery

  1. Encourage new ideas, approaches, and solutions at any time, from any person, on any subject. Permit people to explore.
  2. Have a single repository for all new ideas to be logged, explored, collaborated, prioritized, and developed.
  3. Consider designating time per week or month for people to explore, discover, prove, or disprove ideas and assert priority.
  4. Consider having regular lunch and learns or spontaneous blitzkrieg meetings whereby authors present and explore their problem statements and innovations collaboratively.
  5. Have teams create and review value propositions to receive formal funding.
  6. Use Go/No-Go decision milestones during the funding period.
  7. Encourage folks to explore their ideas on their own time or designate hours per week or month where folks are free to experiment with their ideas.
  8. Define funding parameters within which each effort must exist to manage time, risk, security, and spend.
  9. Know when to say when.
  10. Set and manage expectations with your uplines and peers.

We are all required to change, adapt, and evolve regularly. Exploratory thinking and testing is an enabler. The CTO did the right thing investing in the future and trusting his people. And the Lt. did the right thing working to discover options for the future. They were both intentional.

However, desired outcomes, operational parameters of the engagement, and go/no-go decision points need to be articulated and agreed upon in advance. Outcomes, spend, and activities must also be intentional. Else, it will only look like a toy box project leaving everyone wondering what value they are receiving for the spend.

Categories
Leadership

Approaches to Leadership: Hurricanes, Wakes, Ripples and Deserts

This article was previously published on LinkedIn.

Leadership styles make or break companies. Why?

Teams reflect leadership.

It doesn’t take long to look in the mirror. If you don’t have a mirror, it only takes a cup of coffee to ask some teammates to reflect on your history working with people, projects, teams, and companies.

If you’re dissatisfied with your company, teams, projects, progress, and output, look in the mirror first. It could be you

Hurricane Leaders

Hurricane Leadership

Erratic, unpredictable, contrarian, immeasurable collateral damage

No one sees you coming. No one knows where you’re going. Teams see you’re on a rampage, but don’t know how to help. So, they stay out of the way and otherwise ensure damage to themselves and their interests is minimized – hoping you don’t swing back around for another go. The general assumption when you are present is a higher probability of storms with damage than rain enabling growth.

Your presence breaks things, sometimes people.

Leaders Who Leave Wakes

Leaders Who Leave Wakes

Purposed, directional, controlled, consistent, enabling

People see where you’re going and want to be part of the journey. And because of your plan, communication, and choices, teams who journey with you can be part of an experience that changes in all directions.

Because of you, those who journey with you can make use of the wake you leave behind in a constructive, growing, and useful manner. Your wake changes the company, culture, people, and teams.

Leaders Who Leave Ripples

Leaders Who Leave Ripples

Modest, enjoyable, optional, a lovely time

Who doesn’t love a pleasant afternoon aboard a boat with friends? The gentle hum of the motors barely noticeable, a warm, gentle breeze, the time just between a hot afternoon and dark evening, both without clouds. Sunset.

Leaders who want a sunset cruise with chums enable others to relax. They need only aspire to the level exemplified. No need to rock the boat, make big waves, get out there and jump wake walls, increase skills, or otherwise even get out of the boat. This leadership style is peaceful. People like to be with you knowing that the demands may be low.

Your ripples eventually make contact with other parts of the ecosystem. To what extent change is stimulated, at what velocity or frequency may be variable, even unnoticed.

Desert Leaders

Deserts Leadership: Not Present

When there is no vision, mission, set of objectives, or desired outcomes, people do not know why to gather, what to work on together or even why they, or the company, matter. Leaders are looked upon to cast a vision, rally the people, lead the charge, and foster culture. In the absence of present, active leadership, folks tend to look for someone else, somewhere else, where they are recognized, valued, a part of something bigger than themselves.

Your impact on the culture, team, project, and company is negligible. There are no hurricanes, wakes, or ripples. Nothing is happening as a result of your leadership.

How Do You Choose Your Leadership Impact?

  1. Know what problem you want to solve
  2. Know what desired outcome you want after solving the problem
  3. Plan your leadership approach to foster desired change
  4. Communicate the what and the why of the problem and outcome
  5. Include the teams in solutioning, structuring, and implementation
  6. Continually adapt your leadership style contextually

What type of teams do you have in your company? They usually reflect leadership.


I’ve made a commitment to write more articles in 2020. It is material we discuss every day at Trility and with our clients. If you’d like to keep informed and even interact, please connect or follow me on LinkedIn. Or we can send you an email

We are also always looking for system thinkers to join us – those who can see the larger landscape and do the work as well.

Categories
Operational Modernization

A Basis for Automation, Part 2: Inclusive Continuous Delivery

This article was previously published on LinkedIn.

You are the Delivery Manager. There are 1,000 people in your organization. Twenty-five of them are on your project. Of the 25, six defined “done” for this deliverable. In your opinion, you delivered some of your best work. Twelve people, not on your project, come out of the woodwork telling you things are missing from the deliverable. You’re surprised. The project sponsor is mad. Your big boss wants answers. And the delivery team wonders why you’re letting people say harsh things about the deliverable and team without a fight. The deliverable cannot go to market until you get this figured out. What happened?

Many people suggest this is a problem with the Delivery Manager. Some suggest that stakeholder requirements were not identified and managed correctly. Others blame tools, processes, procedures; some notice internal kingdom-building, protectionism, and turf wars. A few even suggest the technologists who built the system are basement-dwelling anti-social figures of darkness who deliver what they want and don’t seem to know or care about business needs. While these conversations may be entertaining, in the end, finger-pointing conversations don’t solve the real problem.

  • How does it even happen that we miss stakeholders and requirements?
  • How do we ensure we’ve identified and included all pertinent stakeholders in project definition, verification, and validation?
  • How are we going to deliver all these additional requirements, definitions of done, and system attributes?

The answer is simple and accessible.

How Do We Miss Stakeholders and Requirements?

The problem starts inconspicuously at the beginning of the project when people are classified as one of two groups: requesters and builders.

Software Builders vs. Software Requesters
Figure 1. Projects are set up to differentiate “us” and “them.”

After that, requesters are decomposed further into three groups – now, later, never.

  • We’ll concentrate on these roles/users/user classes now,
  • We’ll consider those other roles/users/user classes later, and
  • We’re not going to consider these roles/users/user classes
Backlog Prioritization in Software Development
Figure 2. Groomed backlogs manage scope, priority, risk, time and cost

From above, the “deliver now” requests are then prioritized even further into:

  • We’ll do these things first,
  • We’ll do those other things immediately thereafter, and
  • We’ll do these other things later.

It seems deductively logical and very efficient.

Grooming a backlog is a learned skill. And when done well, it enables team focus and success, manages project risk and reward, as well as provides iteration upon iteration of value to the targeted stakeholders. A single, frequently curated, and prioritized backlog is a healthy core behavior of successful projects.

Over-zealous backlog grooming may also be why you’re missing expected functions, features, and attributes when projects are delivered. The project team thinks they are done, and yet things are missing.

Have you considered that teams are so efficient at curating the backlog that stakeholders and requirements are collateral damage?

How Do We Make Sure We’ve Included Everyone Necessary?

Create a cross-functional strike team responsible to securely operationalize the planned project deliverable – from inception to delivery. A project is not just documents and code. Projects operationalize ideas to serve clients, expand opportunities for growth, and generate revenue.

To securely operationalize requires a complete end-to-end cast of people from project sponsorship to design, develop, test, inspect, and secure to back-office operations including finance, procurement, sales management, and support services.

Cross-Functional Strike Team, (c) Trility, LLC
Figure 3. Cross-Functional Strike Teams are special-purpose teams designed to discover the superset of requirements ensuring common knowledge, purpose, priority, decision, investment, and commitment.

Then as a team, perform the following steps incrementally and iteratively through the course of the project – and support – efforts:

1. Identify the problem statement and desired outcomes.

2. Identify the work and definitions of done per work item. Focus upon roles, capabilities, and attributes.

3. Structure the work into independently digestible chunks with clear definitions of done. Note: Common today is the use of Epics, User Stories, and Acceptance Criteria.

To have the highest probability of delivering something useful to most, we need to know what they want by including more people, more roles. Conversely, to have the highest probability of dissatisfaction with the deliverable, exclude people and guess on their behalf.

You may be wondering, if there are going to be higher numbers of stakeholders and higher amounts of requirements, how are we going to get this done without lengthening the project, increasing the cost and adding to staff?

Enter the role of automation.

What Is Automated Continuous Delivery?

Developers are expected to do many things. Writing code is a means to an end, not the deliverable itself. To deliver more value, minimize context-switching, and increase flow, developers seek optimizations.

For some, optimizing means cutting things out and kicking them down the road. For many others, optimizing means automation.

What do I have to do?

What can I aggregate, separate or eliminate

What do I have to do manually?

What can I automate?

Continuous Integration, Continuous Delivery
Figure 4. An example of a continuous flow experience for a developer.

A fantastic result of developers who seek efficiencies through automation is that those same efficiencies can be used all along the delivery pipeline and out into general availability/production.

For some great examples of how developers seek to simplify through automation, look to Will Rubel, and his four-part conversation on test automation in delivery pipelines.

Figure 5. An example of continuous flow at the (team, project, program, division, enterprise) release level.

What about all of this makes automation and continuous delivery inclusive?

Remember the cross-functional strike team previously discussed? Many times, stakeholder numbers are minimized to control scope, time, cost, risk, and complexity because the developer, respectfully, is the constraint.

Automated continuous delivery pipelines change the math regarding how many stakeholders, requirements, and developers are required to deliver value.

With automation, we decouple the dependency between requirements and the capacity to meet them.

How Will We Deliver More Value with Fewer People?

There are three keys to moving into this new paradigm:

  • Use your cross-functional teams to define work.
  • All work must have definitions of done articulated as acceptance criteria.
  • All acceptance criteria become automated tests that are applied to user stories on the continuous delivery pipeline.
Strike teams build automated tests/checks in the continuous delivery pipeline. (c) Trility, LLC
Figure 6. Strike teams build automated tests/checks which are included in the continuous delivery pipeline designed to prove or disprove a user story is still behaving as expected.

Cross-functional teams contribute to the discovery, definition, and prioritization of roles, capabilities, and attributes in the backlog and participate in demos to verify value. The smaller strike team who makes the code commit, builds, verifies, and delivers the solutions also creates automated tests along the way.

Together, they rely upon definitions of done, automation, and the continuous delivery pipeline. Together they deliver a solution.

Principles for Delivering the Solution Everyone Wanted

  1. Let anyone at any time, for any reason, add any ideas, suggestions, or requests to the backlog.
  2. Curate, cultivate and prioritize the backlog perpetually, daily
  3. Use cross-functional delivery teams for backlog prioritization, epic and user story build-out, design, development, inspection, testing, and delivery behaviors from beginning to end of the project.
  4. Tell teams to focus on flow.
  5. Use version-controlled, continuous delivery principles from end to end of the project, i.e., from developer out into product implementation support and evolution (General Availability (GA)).
  6. Automate. Automate. Automate. If you would like to further explore the reasons and value behind software automation, refer to A Basis for Automation, Part I.

We don’t need to run from, defend against or hide from stakeholders and their expectations. Instead, invite them. Then continuously curate, prioritize, and automate their feedback.

How Do I Get Started?

If you’re perplexed by delivery teams saying they are done, but the solution delivered seems short on features, functions, and value, the conversation more likely lays around how software is being defined and delivered instead of what.

  • Ask your teams how they choose stakeholders.
  • Ask your teams how they choose requirements.
  • Ask your teams if, how and to what extent they use continuous delivery principles and automation. Do they pursue continuous flow?

Their answers point you to your next steps.

Interested in Examining Flow?

Trility joined Bâton Global for a podcast series on Human-Centric Transformation. In the four-part series, they discuss how leaders and technology can simplify and automate processes for team members, stakeholders, and customers.

Before you listen, view our companion infographics highlighting key takeaways as these organizations discuss how to better respond to industry headwinds and pressures.

Categories
News

Remote Work Maturity Model: Where is Your Company During Coronavirus Pandemic?

The Internet is full of tips for “remote workers.” However, if you are a business leader who is responsible for enabling remote work, this Maturity Model can help you identify gaps you need to address and communicate timely decisions.

This Remote Work Maturity Model includes these four areas: Policies, Technology, Secured Technology, and ultimately maintaining Operational Productivity.

Having Policies & Technology in Place

Having the appropriate policies and technology is the first step in the Remote Work Maturity Model. You have policies that outline expectations and required behaviors, and all or some of your team may have laptops or devices that allow them to work from anywhere. Many companies have not yet needed to solve either of these problems and have been caught off-guard with the immediacy of COVID-19 pandemic. If you have both, now is a good time provide clarity on those expectations through communication.

Remote Work Maturity Model

Ensuring Security Measures are in Place

Having a portion or all of your workforce operating remotely adds new attack vectors that your information security team needs to address. Depending on applications, networks, and other solutions you have in place, adjustments might need to be made to existing security solutions. You might even realize you need to implement stronger, more stringent security measures. And again, re-educating your workforce about phishing, physically securing their devices, and avoiding malicious websites is critical. 

It’s also a good time to confirm if you have tested, secured technology solutions in place to prevent attacks. Encrypted devices? Redundant network connections? VPN solutions? Secure cloud configuration? Traffic monitoring? Logging? Alerting?

Remote Work Maturity Model

Keeping People Productive

Once you’ve confirmed policies and tested, secured technology solutions are in place, your focus is on maintaining the same level of operational productivity. Coordinating work and collaboration across teams, departments, and the entire organization presents an even greater challenge when dynamic changes occur this quickly. Teams must understand objectives, along with the necessary resources and collaboration tools to optimize engagement and productivity while working remotely. 

How you respond to the novel Coronavirus in keeping your team safe, productive, and secure, can serve as an experiment where your teams have the chance to discover new, more productive ways to work together.

Need Sound Advice Now?

The Trility team wants to equip you and your team to quickly adapt in order for your people to remain safe, productive, and secure.

Equipping you to keep people safe, productive, secure

We are a geographically distributed company with remote workers and clients located all over the United States and even internationally. As companies navigate these unprecedented times, we are offering “free-of-charge” conversations. If you need a sounding board for some of these gaps, we will do our best to connect you to the experts on our team.

Categories
News

Maintaining Business Continuity Amid Epidemic Response

After a tumultuous week which concluded with President Trump declaring a national emergency in response to novel coronavirus concerns, companies have been enacting their business continuity plans and instructing their employees to work remotely, some for the first time.  As a company who has enabled remote-work since our founding, we wanted to share how we make certain our people work remotely in the most effective and secure manner. We offer these lessons learned to help those mitigating these disruptions and challenges.

How We Do Business Securely

Live Anywhere, Work Anywhere

Our employees work from home but also work where our clients need them. This means we understand how to work securely from home, our office, and our clients’ offices. In normal circumstances, these include hotels, sky clubs, coffee shops. If you have questions or concerns, we can help. 

Cloud-Based Availability

We help companies securely migrate to the cloud and it’s been our home since the beginning. Everything we utilize for internal operations and client-facing business is cloud-based – allowing Trility to seamlessly operate from anywhere with no disruption. Cloud security and reliability are a No. 1 priority to us. With the majority of breaches in 2019 due to cloud storage misconfigurations, we must continue to be diligent and never compromise when it comes to security.

Secure Device Management

As businesses transition to work from home, reinforce device management policies, and determine if they are secure enough. Our team is geographically distributed across the country. As a best-practice, consider some of these approaches we use:

  • Hard drives must have full disk encryption
  • Screens must be locked when you are away from the keyboard
  • Screens must be set to auto-lock after five minutes if you are away from the keyboard and forget to lock
  • Do not store customer data locally to the laptop
  • Don’t attach your laptop to insecure public wireless networks – every time it is available, use VPN
  • Use modern operating systems and tools and keep them current at all times

Lines of Communication

The days of operational 8-5 hours are gone. Maintain several lines of communication in order to meet the needs of clients, partners, and employees. We use Slack, Zoom, Microsoft Teams, Confluence, JIRA, GSuite (Google Productivity Suite), and several more to align with our client’s preferred use of communication. Find one that works for your business.

  • Have dynamic communication tools to serve as a foundation for most written communications.   This is preferably not email. 
  • Use communication tools that allow for one-to-one, group, team, and enterprise-wide communications through various channels – video and voice-capable ones are ideal.  
  • Provide a stream of news and updates to employees throughout the organization.
  • Integrate the communication tool via APIs to your various productivity tools like your project management, ERP, CRM, or issue tracking systems.  
  • For dynamic document collaboration, use cloud-based tools with traceable version control.

Redundant Systems

No one person or location holds information, and backups are always in place. Whether it’s a person, a process, or line of code, we have systems and backups for completing work and documentation is always provided because we never want our clients handcuffed to us.

Start Simple. Then Automate. And Always Be Secure.

Under normal circumstances, we help companies defend or extend their market share in an era of rapid disruption by simplifying, automating, and securing each iteration. The era of disruption has been disrupted. We are here to help.

The Internet is full of noise right now. Big problems need to be solved, and they can’t all be Googled.

Need a Sounding Board?

If you are in need of sound advice, reach out to our team. If you aren’t sure who to contact, email marketing@trility.io and they will direct you to the right person.

About Trility

For those wanting to defend or extend their market share in an era of rapid disruption, Trility simplifies, automates, and secures the journey and has a proven history of reliable delivery results. Headquartered in Des Moines, Iowa, with teams in Omaha, Neb., and Chicago, Ill., our people live everywhere and work where needed to help clients navigate their business evolution with certainty.

Categories
Cloud & Infrastructure

Part IV: Complex Practical Examples of DevOps Unit Testing

In my previous article, I provided a simple example of mocking an AWS resource using localstack, and testing with the python terraform-compliance module. In this example, I will provide a more extensive example using kitchen-terraform and terraform-compliance to deploy the following resources in AWS us-east-1 and us-west-2 regions.

  1. VPC
  2. Subnet
  3. Internet Gateway
  4. Route Table
  5. Route Table Association
  6. Security Group
  7. Key Pair
  8. 2 X EC2 Instance

To begin this example, you will need the following:

  1. Terraform 
  2. Ruby
  3. Python3
  4. Python3 virtualenv module
  5. An AWS account with credentials configured in ~/.aws
  6. An AWS role or user with at least the minimum permissions:
{
 "Version": "2012-10-17",
 "Statement":
   [
     {
       "Sid": "Stmt1469773655000",
       "Effect": "Allow",
       "Action": ["ec2:*"],
       "Resource": ["*"]
     }
   ]
}

Next, we need to set up a Python3 virtual environment, activate the environment and install the python terraform-compliance module.

which python3
/Library/Frameworks/Python.framework/Versions/3.8/bin/python3
cd ~
mkdir virtualenvs
cd virtualenvs
virtualenv terraform-test  -p /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
source terraform-test/bin/activate
pip install terraform-compliance

Now, we need to create a projects directory and download the sample code from github.

cd ~
mkdir projects
cd projects
git clone git@github.com:rubelw/terraform-kitchen.git
cd terraform-kitchen

Now we are ready to run our tests, by executing the ‘execute_kitchen_terraform.sh’ file.

This script will perform the following functions:

  1. Install bundler
  2. Install required gems
  3. Create public and private key pair
  4. Initialize terraform project
  5. Test terraform plan output against terraform-compliance features
  6. Execute kitchen test suite
  • kitchen destroy centos(us-east-1)
  • kitchen create centos(us-east-1)
  • kitchen converge centos(us-east-1)
  • kitchen verify centos (us-east-1)
  • kitchen destroy centos(us-east-1)
  • kitchen destroy ubuntu(us-west-2)
  • kitchen create ubuntu(us-west-2)
  • kitchen converge ubuntu(us-west-2)
  • kitchen verify ubuntu(us-west-2)
  • kitchen destroy ubuntu(us-west-2)
./execute_kitchen_terraform.sh

This script will begin by checking if bundler is installed, and then installing the necessary ruby gems.

Successfully installed bundler-2.1.4
Parsing documentation for bundler-2.1.4
Done installing documentation for bundler after 2 seconds
1 gem installed
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies..
…
Using kitchen-terraform 5.2.0
Bundle complete! 1 Gemfile dependency, 185 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

Next the script will test if the public/private keypair exists in the test/assets directory, if not, it will create the key pair.

checking if test/assets directory exists
Generating public/private rsa key pair.
Your identification has been saved in test/assets/id_rsa.
Your public key has been saved in test/assets/id_rsa.pub.
The key fingerprint is:
SHA256:0oryWP5ff8kBwQPUSCrLGlVMFzU0rL7TQtJSi6iftyo Kitchen-Terraform AWS provider tutorial
The key's randomart image is:
+---[RSA 4096]----+
|       ooo*X=    |
|       ..o. *o   |
|      o .  . o   |
|     o +  o .    |
|    . +.S= . .   |
|     +.o+ =   .  |
|  . +..  +.o . o |
|   *E  ...+.. +  |
|  . o+=+o. o..   |
+----[SHA256]-----+

Next, the script will test the terraform project, using the python terraform-compliance module, and features located in test/features.

The script begins by testing if the terraform project has been initialized, and if not, initializing the project.

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.1.2...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

After terraform initialization, the script will execute ‘terraform plan’ and output the plan in json format. It will then test the terraform output against the features in the test directory.

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.reachable_other_host will be created
  + resource "aws_instance" "reachable_other_host" {
      + ami                          = "ami-1ee65166"
      + arn                          = (known after apply)
      + associate_public_ip_address  = true
      + availability_zone            = (known after apply)
…
Plan: 11 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: myout

To perform exactly these actions, run the following command to apply:
    terraform apply "myout"

terraform-compliance v1.1.11 initiated

🚩 Features	: /terraform-kitchen/test/features
🚩 Plan File	: /terraform-kitchen/myout.json

🚩 Running tests. 🎉

Feature: security_group  # /terraform-kitchen/test/features/security_group.feature
    In order to ensure the security group is secure:

    Scenario: Only selected ports should be publicly open
        Given I have AWS Security Group defined
        When it contains ingress
        Then it must only have tcp protocol and port 22,443 for 0.0.0.0/0

1 features (1 passed)
1 scenarios (1 passed)
3 steps (3 passed)

You may be asking, why do we need both terraform-compliance features and kitchen-terraform fixtures for our testing? The purpose of terraform-compliance features is to have a repository of global, enterprise-level features and tests, which get applied to all projects. For example, the test displayed above will test security groups, so only ports 22 and 443 are open. No other ports should be open in the security group.

The kitchen-terraform fixtures and tests are designed for unit testing a single terraform project, and are not to be applied to every terraform project. 

Continuing with the script execution, the script will now run the kitchen-terraform tests. It begins by attempting to destroy any existing terraform state in the applicable region.

-----> Starting Test Kitchen (v2.3.4)
-----> Destroying <complex-suite-centos>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.12.21
       + provider.aws v2.51.0
       + provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
       Initializing modules...
       
       Initializing the backend...
       
       Initializing provider plugins...
       
       Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Selecting the kitchen-terraform-complex-suite-centos Terraform workspace...
$$$$$$ Finished selecting the kitchen-terraform-complex-suite-centos Terraform workspace.
$$$$$$ Destroying the Terraform-managed infrastructure...
       module.complex_kitchen_terraform.random_string.key_name: Refreshing state... [id=none]
…
       Destroy complete! Resources: 11 destroyed.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Selecting the default Terraform workspace...
       Switched to workspace "default".
$$$$$$ Finished selecting the default Terraform workspace.
$$$$$$ Deleting the kitchen-terraform-complex-suite-centos Terraform workspace...
       Deleted workspace "kitchen-terraform-complex-suite-centos"!
$$$$$$ Finished deleting the kitchen-terraform-complex-suite-centos Terraform workspace.
       Finished destroying <complex-suite-centos> (3m31.75s).
-----> Test Kitchen is finished. (3m32.88s)

The script will then initialize the terraform working directory and select a new terraform workspace.

-----> Starting Test Kitchen (v2.3.4)
-----> Creating <complex-suite-centos>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.12.21
       + provider.aws v2.51.0
       + provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
       Upgrading modules...
       - complex_kitchen_terraform in ../../..
       
       Initializing the backend...
       
       Initializing provider plugins...
       - Checking for available provider plugins...
       - Downloading plugin for provider "random" (hashicorp/random) 2.1.2...
       - Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0...
       
       Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Creating the kitchen-terraform-complex-suite-centos Terraform workspace...
       Created and switched to workspace "kitchen-terraform-complex-suite-centos"!
       
       You're now on a new, empty workspace. Workspaces isolate their state,
       so if you run "terraform plan" Terraform will not see any existing state
       for this configuration.
$$$$$$ Finished creating the kitchen-terraform-complex-suite-centos Terraform workspace.
       Finished creating <complex-suite-centos> (0m16.81s).
-----> Test Kitchen is finished. (0m17.97s)

The next step in the script is to run the ‘kitchen converge’.  This step will converge the platforms in the kitchen.yml file.

-----> Starting Test Kitchen (v2.3.4)
-----> Creating <complex-suite-centos>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.12.21
       + provider.aws v2.51.0
       + provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
       Upgrading modules...
       - complex_kitchen_terraform in ../../..
       
       Initializing the backend...
       
       Initializing provider plugins...
       - Checking for available provider plugins...
       - Downloading plugin for provider "random" (hashicorp/random) 2.1.2...
       - Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0...
       
       Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Creating the kitchen-terraform-complex-suite-centos Terraform workspace...
       Created and switched to workspace "kitchen-terraform-complex-suite-centos"!
       
       You're now on a new, empty workspace. Workspaces isolate their state,
       so if you run "terraform plan" Terraform will not see any existing state
       for this configuration.
$$$$$$ Finished creating the kitchen-terraform-complex-suite-centos Terraform workspace.
       Finished creating <complex-suite-centos> (0m16.81s).
-----> Test Kitchen is finished. (0m17.97s)

Finally, the script will execute ‘kitchen verify’ to test the deployed project against the test suite.

-----> Starting Test Kitchen (v2.3.4)
-----> Setting up <complex-suite-centos>...
       Finished setting up <complex-suite-centos> (0m0.00s).
-----> Verifying <complex-suite-centos>...
$$$$$$ Reading the Terraform input variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform input variables from the Kitchen instance state.
$$$$$$ Reading the Terraform output variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform output variables from the Kitchen instance state.
-----> Starting verification of the systems.
$$$$$$ Verifying the 'local' system...

Profile: complex kitchen-terraform (complex_suite)
Version: 0.1.0
Target:  local://

  ✔  state_file: 0.12.21
     ✔  0.12.21 is expected to match /\d+\.\d+\.\d+/
  ✔  inspec_attributes: static terraform output
     ✔  static terraform output is expected to eq "static terraform output"
     ✔  static terraform output is expected to eq "static terraform output"


Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped
Test Summary: 3 successful, 0 failures, 0 skipped
$$$$$$ Finished verifying the 'local' system.
…
$$$$$$ Finished verifying the 'remote' system.
$$$$$$ Verifying the 'remote2' system...
DEPRECATION: AWS resources shipped with core InSpec are being moved to a resource pack for faster iteration. Please update your profiles to depend on git@github.com:inspec/inspec-aws.git . Resource 'aws_vpc' (used at /private/tmp/terraform-kitchen/test/integration/complex_suite/controls/aws_resources.rb:11)
DEPRECATION: AWS resources shipped with core InSpec are being moved to a resource pack for faster iteration. Please update your profiles to depend on git@github.com:inspec/inspec-aws.git . Resource 'aws_subnets' (used at /private/tmp/terraform-kitchen/test/integration/complex_suite/controls/aws_resources.rb:16)
DEPRECATION: AWS resources shipped with core InSpec are being moved to a resource pack for faster iteration. Please update your profiles to depend on git@github.com:inspec/inspec-aws.git . Resource 'aws_security_group' (used at /private/tmp/terraform-kitchen/test/integration/complex_suite/controls/aws_resources.rb:22)

Profile: complex kitchen-terraform (complex_suite)
Version: 0.1.0
Target:  aws://

  ✔  aws_resources: VPC vpc-00aa64d66abfa8e9c
     ✔  VPC vpc-00aa64d66abfa8e9c is expected to exist
     ✔  VPC vpc-00aa64d66abfa8e9c cidr_block is expected to eq "192.168.0.0/16"
     ✔  EC2 VPC Subnets with vpc_id == "vpc-00aa64d66abfa8e9c" states is expected not to include "pending"
     ✔  EC2 VPC Subnets with vpc_id == "vpc-00aa64d66abfa8e9c" cidr_blocks is expected to include "192.168.1.0/24"
     ✔  EC2 VPC Subnets with vpc_id == "vpc-00aa64d66abfa8e9c" subnet_ids is expected to include "subnet-000c991d9264c3a5f"
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to exist
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"198.144.101.2/32", :port=>22}
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"73.61.21.227/32", :port=>22}
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"198.144.101.2/32", :port=>443}
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"73.61.21.227/32", :port=>443}
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f group_id is expected to cmp == "sg-0bcdd1f63ba2a4b6f"
     ✔  EC2 Security Group sg-0bcdd1f63ba2a4b6f inbound_rules.count is expected to cmp == 3
     ✔  EC2 Instance i-0db748e47640739ea is expected to exist
     ✔  EC2 Instance i-0db748e47640739ea image_id is expected to eq "ami-ae7bfdb8"
     ✔  EC2 Instance i-0db748e47640739ea instance_type is expected to eq "t2.micro"
     ✔  EC2 Instance i-0db748e47640739ea vpc_id is expected to eq "vpc-00aa64d66abfa8e9c"
     ✔  EC2 Instance i-0db748e47640739ea tags is expected to include {:key => "Name", :value => "kitchen-terraform-reachable-other-host"}


Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 17 successful, 0 failures, 0 skipped
$$$$$$ Finished verifying the 'remote2' system.
-----> Finished verification of the systems.
       Finished verifying <complex-suite-centos> (0m43.58s).
-----> Test Kitchen is finished. (0m44.76s)

The last step in the script is the ‘kitchen destroy’.  This will destroy all AWS resources instantiated for the test.

-----> Starting Test Kitchen (v2.3.4)
-----> Destroying <complex-suite-centos>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.12.21
       + provider.aws v2.51.0
       + provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
       Initializing modules...
       
       Initializing the backend...
       
       Initializing provider plugins...
       
       Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory
…
       module.complex_kitchen_terraform.aws_vpc.complex_tutorial: Destroying... [id=vpc-00aa64d66abfa8e9c]
       module.complex_kitchen_terraform.aws_vpc.complex_tutorial: Destruction complete after 1s
       
       Destroy complete! Resources: 11 destroyed.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Selecting the default Terraform workspace...
       Switched to workspace "default".
$$$$$$ Finished selecting the default Terraform workspace.
$$$$$$ Deleting the kitchen-terraform-complex-suite-centos Terraform workspace...
       Deleted workspace "kitchen-terraform-complex-suite-centos"!
$$$$$$ Finished deleting the kitchen-terraform-complex-suite-centos Terraform workspace.
       Finished destroying <complex-suite-centos> (2m47.02s).
-----> Test Kitchen is finished. (2m48.17s)

Now the scripts will perform the same steps with ubuntu instances in us-west-2 region.

Future of Infrastructure Testing and Standards

In summary, I hope you have enjoyed this four-part series regarding infrastructure testing.  While these articles only covered specific situations and scenarios for infrastructure testing and deployments, I hope it causes your organization to open a discussion about the future direction of infrastructure testing and standards.

Read the Entire DevOps Testing Series


Categories
Cloud & Infrastructure

Part III: Practical Examples of DevOps Unit Testing

In my last two articles, I’ve talked conceptually and theoretically about the need for DevOps testers.

Part I: Does DevOps Need Dedicated Testers?
Part II: 2019 Cloud Breaches Prove DevOps Needs Dedicated Testers

In this article, I will provide practical examples of unit testing.

Since public cloud storage seems to be a common problem, I will begin with an example unit test for a terraform project which creates a simple S3 bucket.

First, we need to install localstack, so we can test AWS locally.

pip install localstack
export SERVICES=s3
export DEFAULT_REGION='us-east-1'
localstack start

In a new console/terminal and new directory, create a simple terraform project. The provider.tf file should point to the localstack ports.

provider "aws" {
	region = "us-east-1"
	skip_credentials_validation = true
	skip_metadata_api_check = true
	s3_force_path_style = true
	skip_requesting_account_id = true
	skip_get_ec2_platforms = true
	access_key = "mock_access_key"
	secret_key = "mock_secret_key"
	endpoints {
    	s3 = "http://localhost:4572"
	}
}

resource "aws_s3_bucket" "b" {
  bucket = "test"
  acl    = "private"

  tags = {
	Name    	= "My bucket"
	Environment = "Dev"
  }
}

Deploy the terraform project.

terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_s3_bucket.b: Refreshing state... [id=test]

------------------------------------------------------------------------


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.b will be created
  + resource "aws_s3_bucket" "b" {
  	+ acceleration_status     	= (known after apply)
  	+ acl                     	= "private"
  	+ arn                     	= (known after apply)
  	+ bucket   	               = "test"
  	+ bucket_domain_name      	= (known after apply)
  	+ bucket_regional_domain_name = (known after apply)
  	+ force_destroy           	= false
  	+ hosted_zone_id          	= (known after apply)
  	+ id                          = (known after apply)
  	+ region                  	= (known after apply)
  	+ request_payer           	= (known after apply)
  	+ tags                    	= {
      	+ "Environment" = "Dev"
      	+ "Name"	    = "My bucket"
    	}
  	+ website_domain          	= (known after apply)
  	+ website_endpoint        	= (known after apply)

  	+ versioning {
      	+ enabled	= (known after apply)
      	+ mfa_delete = (known after apply)
    	}
	}

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terrafor can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$ terraform apply
aws_s3_bucket.b: Refreshing state... [id=test]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.b will be created
  + resource "aws_s3_bucket" "b" {
  	+ acceleration_status     	= (known after apply)
  	+ acl                     	= "private"
  	+ arn         	            = (known after apply)
  	+ bucket                  	= "test"
  	+ bucket_domain_name      	= (known after apply)
  	+ bucket_regional_domain_name = (known after apply)
  	+ force_destroy           	= false
  	+ hosted_zone_id          	= (known after apply)
  	+ id                      	= (known after apply)
  	+ region                  	= (known after apply)
  	+ request_payer           	= (known after apply)
  	+ tags                    	= {
      	+ "Environment" = "Dev"
      	+ "Name"    	= "My bucket"
    	}
  	+ website_domain          	= (known after apply)
  	+ website_endpoint        	= (known after apply)

  	+ versioning {
      	+ enabled	= (known after apply)
      	+ mfa_delete = (known after apply)
    	}
	}

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions in workspace "kitchen-terraform-base-aws"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_s3_bucket.b: Creating...
aws_s3_bucket.b: Creation complete after 0s [id=test]

Create a test.py file with the following code to test the deployment of the S3 bucket.

import boto3


def test_s3_bucket_creation():
	s3 = boto3.client(
    	's3',
    	endpoint_url='http://localhost:4572',
    	region_name='us-east-1'
	)
	# Call S3 to list current buckets
	response = s3.list_buckets()

	# Get a list of all bucket names from the response
	buckets = [bucket['Name'] for bucket in response['Buckets']]

	assert len(buckets) == 1

Test that the bucket was created.

$ pytest test.py
=============================================================== test session starts ===============================================================
platform darwin -- Python 3.6.0, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
rootdir: /private/tmp/myterraform/tests/test/fixtures
plugins: localstack-0.4.1
collected 1 item

test.py .

Now, let’s destroy the S3 bucket.

$ terraform destroy
aws_s3_bucket.b: Refreshing state... [id=test]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_s3_bucket.b will be destroyed
  - resource "aws_s3_bucket" "b" {
  	- acl                     	= "private" -> null
  	- arn                     	= "arn:aws:s3:::test" -> null
  	- bucket                  	= "test" -> null
  	- bucket_domain_name      	= "test.s3.amazonaws.com" -> null
  	- bucket_regional_domain_name = "test.s3.amazonaws.com" -> null
  	- force_destroy           	= false -> null
  	- hosted_zone_id          	= "Z3AQBSTGFYJSTF" -> null
  	- id                      	= "test" -> null
  	- region                  	= "us-east-1" -> null
  	- tags                    	= {
      	- "Environment" = "Dev"
      	- "Name"    	= "My bucket"
    	} -> null

  	- object_lock_configuration {
    	}

  	- replication_configuration {
    	}

  	- server_side_encryption_configuration {
    	}

  	- versioning {
      	- enabled	= false -> null
      	- mfa_delete = false -> null
    	}
	}

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_s3_bucket.b: Destroying... [id=test]
aws_s3_bucket.b: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.

Next, we will install the terraform-compliance python module.

pip install terraform-compliance

Next, we will set up the directory for our test.

mk features
cd features

Next, make a file named s3.features inside the features directory with the following content.

Feature: test

 	In order to make sure the s3 bucket is secure:

 	Scenario: No public read
     	Given I have AWS S3 Bucket defined
     	When it contains acl
     	Then its value must not match the "public-read" regex

Now, we will return to the root directory for the project and run a terraform plan to get the plans output in json format.

terraform plan -out=myout
terraform show -json myout > myout.json

Lastly, we will test the terraform project against the feature file to see if the project is compliant.

$ terraform-compliance -p /tmp/junk/myout.json -f /tmp/junk/features
terraform-compliance v1.1.7 initiated

🚩 Features : /tmp/junk/features
🚩 Plan File : /tmp/junk/myout.json

🚩 Running tests. 🎉

Feature: test  # /tmp/junk/features/s3.feature
	In order to make sure the s3 bucket is secure:

	Scenario: No public read
    	Given I have AWS S3 Bucket defined
    	When it contains acl
    	Then its value must not match the "public-read" regex

1 features (1 passed)
1 scenarios (1 passed)
3 steps (3 passed)

As you will notice from the results, all tests passed because the S3 bucket deployed is private.

While these are just basic examples, they are intended to demonstrate the concept of unit testing infrastructure-as-code, and testing for various rules.

Read the Entire DevOps Testing Series


Categories
News

Nathan Levis Joins Trility as Senior Sales Engineer

Trility Consulting® is proud to announce Nathan Levis has joined the Trility team as a Senior Sales Engineer. In this role, Levis will help identify and craft solution engagements for clients to simplify, automate, and secure their paths forward.

Leveraging his breadth of technical expertise and an open-minded approach, Levis will focus on building partnerships to ensure organizations defend or extend their market share in an era of rapid disruption.

Nathan Levis Joins Trility as Senior Sales Engineer

A Holistic View of Business

Nathan brings invaluable Cloud, DevOps, Software Engineering, and agile experience, coupled with a keen interest in holistic business impact to help companies deliver on their most important technology-enabled priorities. He will be a great asset for our clients and for the Trility team.”

Brody Deren, Chief Strategy Officer for Trility

Trility’s outcome-based delivery method means clients receive observations, recommendations, and options to iterate for the best, highest-priority outcome. Levis will help build upon this proven approach and ensure we continue to deliver over and over again on our promises – meeting time, budget, and defined scopes that align with business and technical requirements. 

Comprised of technologists and business consultants, Trility helps organizations of all sizes achieve business and technology outcomes while equipping them for the next iteration in these areas of focus:

  • Cloud and Infrastructure
  • Product Design and Development
  • Information Security
  • Data Strategy and Management
  • Internet of Things (IoT)
  • Operational Modernization

About Trility

For those wanting to defend or extend their market share in an era of rapid disruption, Trility simplifies, automates, and secures the journey and has a proven history of reliable delivery results.

Headquartered in Des Moines, Iowa, with teams in Omaha, Neb., and Chicago, Ill., our people live everywhere and work where needed to help clients navigate their business evolution with certainty.

Categories
Cloud & Infrastructure

Part II: 2019 Cloud Breaches Prove DevOps Needs Dedicated Testers

To prove that DevOps needs a tester, you have to look no further than IdentityForce.com‘s biggest breaches of 2019 and review the types of breaches involved, and investigate why they occurred.

If you notice, a large percentage of the breaches were related to misconfiguration of cloud storage, and the lack of multi-factor authentication to access systems.

So who is primarily at fault: development, operations, security, networking, or DevOps?

While there could be many reasons for ‘open’ cloud storage and single-factor authentication to systems, I would suggest these are DevOps-related mistakes, and DevOps failed to: (1) properly test the security configuration of cloud storage prior to deployment, and (2) also failed to set up multi-factor authentication for accessing systems, and scan images for proper authentication to systems.

Last Line of Defense before Deployment is the Continuous Integration/Continuous Delivery

Some may argue that operations, security and/or networking departments are at fault, but the last line of defense before deployment is the Continuous Integration/Continuous Delivery (CI/CD) pipeline, which should include the application of common rule-sets and tests and is primarily the responsibility of DevOps.

Terraform Sentinal, Nexpose, Other Tools

Others will argue, proper CI/CD tools, such as Terraform Sentinal or Nexpose, or setting-up AWS config rules and using OpsWorks will prevent these issues; and they would be partially correct. These tools provide a layer of security and protection which is similar to application vulnerability scanning tools, but they do not replace unit-testing or integration testing.

Unit Testing Ensures Actual Results Meet Expected Ones

The purpose of unit testing is to ensure the actual results match the expected results. Using public cloud storage as an example, the infrastructure project to create the cloud storage should contain unit-tests which include a: 

  1. check for existence 
  2. check authorizations
  3. check security settings 

Upon deployment of the project, the CI/CD pipeline will execute the unit-test, and if passing, perform integration testing.

Integration Testing for Cross-Boundary Access

The purpose of integration testing is to test individual units combined as a group. From an infrastructure perspective, this means the testing of cross-boundary access, permissions, and functionality. Utilize the public cloud storage example, and assuming the cloud storage had a permission to allow another account access to the storage, there would need to be an integration test for an external account to access the cloud storage – but who writes this code, and how do they know they need to write it?

This is where the concept of a DevOps tester is most applicable. Two separate infrastructure projects have been deployed; one for an account which has a dependency on cloud storage in a separate account, and one for cloud storage in a separate account. Ideally, DevOps should have recognized the dependency when creating the account, and created a unit-test that tests the permission of a mocked-up storage account. Someone would then need to write a separate integration test that is run in the CI/CD pipeline upon completion of both deployments.

Managing the inter/intra project dependencies, ordering, and priority of various infrastructure projects could become very overwhelming for DevOps, and is one of the primary reasons a DevOps tester is needed. Currently, I’m only seeing minimal infrastructure unit-testing, and not seeing any coordinated integration testing across infrastructure projects.

Just like when developers first began performing unit and integration testing, they performed these functions themselves. As the need arose, organizations would hire a software tester, and software testers would take more and more of the testing responsibilities; until software testing fully matured. DevOps is no different than normal software development and is still maturing as a concept.

Infrastructure as Code Maturity Will Require Quality Gateways

As Infrastructure As Code becomes the norm, unit-testing and integration testing will become more common. Eventually, we will mature to a point where we are evaluating infrastructure code for code-quality, and preventing deployments that do not meet quality gateways.

The bottom-line: Infrastructure As Code will eventually mature to include unit-tests and integration testing, and become very similar to a normal software development lifecycle. Organizations should begin to further refine their own strategy on how this maturation will occur, and who will be responsible for the infrastructure testing.

In my next article to publish tomorrow, I provide Practical Examples of DevOps Unit Testing

Read the Entire DevOps Testing Series


Categories
Cloud & Infrastructure

Part I: Does DevOps Need Dedicated Testers?

As a DevOps/Cloud Engineering professional, and human being, I will make eight mistakes for every 100 words typed. This means I make hundreds, if not thousands, of mistakes each week. 

So how do I catch my mistakes? I would like to say I write good unit and integration tests for my infrastructure-related code and have over 90 percent code coverage, but this would be a lie. 

In fact, if you’re like most DevOps and cloud engineering professionals, you are not expected to write unit and integration tests and will rely on external tools to test the infrastructure-related errors. So, why aren’t the same unit and integration testing procedures, which are applied to application code, being applied to infrastructure code?

So, why aren’t the same unit and integration testing procedures, which are applied to application code, being applied to infrastructure code?

While the infrastructure team can utilize resources like Terraform, localstack and Terraform-compliance to mock and test resources, they can not mock the platform and services which will live within the infrastructure. Thus, infrastructure teams will do actual deployments to the development environment, in order to test their infrastructure.

Unfortunately, from a developer-perspective, the development environment is ‘production’, and is expected to be stable, and always available. Developers do not want downtime because the infrastructure team is deploying and testing an infrastructure change – and breaks something.

So, how do we resolve this conflict, in the simplest way possible (assuming the development environment is used 24 hours per day)?

I’ve had good results utilizing the same software testing strategy utilized for applications,  for the infrastructure code-base.

By having infrastructure-related unit and integration tests written and tested against the infrastructure code prior to deployment to a development environment, you can ensure infrastructure changes will not break the development environment.

Infrastructure Unit tests might include:

  1. Testing the resource is created and has the proper parameters
  2. Testing pipeline logic to handle exceptions

Infrastructure Integration tests might include:

  1. Testing connectivity
  2. Testing security
  3. Testing permissions

Application/Platform/Service integration tests might include:

  1. Testing Network Access Control Lists
  2. Testing Security Groups
  3. Testing Route Tables
  4. Testing Permissions
  5. Testing for infrastructure controlled keys
  6. Testing for shared resources, and access to shared resources

Writing Good Tests Requires Infrastructure, Architectural Knowledge 

While development software testers could write Application/Platform/Service tests, they may not have the infrastructure and architectural knowledge to understand how to write good tests. Instead, a DevOps Software Tester team should be responsible for coordinating with all development software testers for infrastructure-related integration tests.

The infrastructure-related integration tests would then become part of the infrastructure deployment pipeline.

For example, before any infrastructure-related changes are deployed to the ‘development’ environment, the infrastructure should be deployed to a new environment and validated. Once all tests are passing, then the infrastructure is deployed. In addition, like with application code, infrastructure code should have at least 90 percent code coverage for all infrastructure resources, contain good infrastructure-related integration tests, and have  90 percent coverage for application-related integration tests.

While this solution does not guarantee an outage to your development environment, it applies a consistent, organizational-wide testing strategy for all code, and should help to catch many of the infrastructure-related coding mistakes.

It also provides an additional career path for software testers to enhance their knowledge and skills, and teach DevOps and cloud engineers how to do proper testing.

Or, you can continue to deploy infrastructure-as-code and not write any unit or integration tests.

Read the Entire DevOps Testing Series

To further support this growing need, I will publish three more articles in the coming days.