Skip to content
BranchesTags

Repository files navigation

build-retorch Quality Gate Status Maven Central (annotations)

RETORCH: Resource-aware End-to-End Test Orchestration

This repository contains a series of components of the RETORCH End-to-End (E2E) test orchestration framework. It's primary goal is to optimize E2E test execution by reducing both the execution time and the number of unnecessary Resource1 redeployment's.

NOTE: The repository is a work in progress, the initial version only made available the annotations, and currently we're migrating the orchestration generator module. Additional components will be added in future releases.

Contents

Quick-start

RETORCH Framework Model

The diagram below shows the RETORCH data model across its four layers: the test annotation layer, the orchestration model, the cloud infrastructure configuration, and the usage profiling artefacts.

flowchart LR
  %% Classifiers
  classDef blue fill:#1e497c,stroke:#000000,color:#ffffff;
  classDef green fill:#234723,stroke:#000000,color:#ffffff;
  classDef red fill:#911111,stroke:#000000,color:#ffffff;
  classDef yellow fill:#ffff00,stroke:#000000,color:#000000;
  classDef orange fill:#f79646,stroke:#000000,color:#ffffff;
  classDef lightBlue fill:#4f81bd,stroke:#000000,color:#ffffff;
  classDef lightGreen fill:#548235,stroke:#000000,color:#ffffff;

  %% Nodes
  A[E2E TEST SUITE]:::purple --> B[Custom Annotations]:::orange
  B --> C[RETORCH Classifier]:::lightBlue

  %% Retorch Section
  subgraph Retorch
    direction TB
    C --> D[Resource Info.]:::lightBlue
    D --> E[RETORCH Aggregator]:::blue
    E --> F[TGroups]:::blue
    F --> G[RETORCH Scheduler]:::blue
    G --> H[TJob]:::blue
    H --> I[RETORCH Orchestrator]:::blue
    I --> J[Execution Plan]:::lightBlue
  end

  %% Resource Section
  subgraph Resource
    direction TB
    C --> K[Resource]:::lightBlue
    K --> L[Access Mode]:::lightBlue
    K --> M[Resource Instance]:::lightBlue
    M --> N[Minimal Capacities]:::lightBlue
    N --> O[Capacity]:::lightBlue
    J --> P[Capacity]:::lightBlue
  end

  %% Cloud Section
  subgraph Cloud
    direction TB
    Q[COI Selection]:::yellow --> R[Cloud Object Instance]:::lightGreen
    R --> S[Cloud Object]:::lightGreen
    R --> T[Contracted Capacities]:::lightGreen
    T --> U[Billing Option]:::lightGreen
    T --> V[Cloud Configuration]:::lightGreen
    V --> W[Profile Generator]:::green
    W --> X[RAW Usage Profile]:::green
    W --> Y[AVG Dataset]:::green
    X --> Z[Profile Plotter]:::green
    Z --> AA[Usage Profile]:::green
    W --> BB[RETORCH DataTuple]:::green
    Y --> CC[Dataset Generator]:::green
  end

  %% Estimation Section
  subgraph Estimation
    direction TB
    CC --> DD[Estimator]:::red
    DD --> EE[Costs]:::orange
  end

  %% Execution Data
  J --> FF[Exec. Plan. Dataset]:::yellow
  FF --> Y
  FF --> W
Loading

RETORCH Annotations

The RETORCH framework provides a set of custom annotations to define and manage Resources used in end-to-end testing. These annotations allow testers to group, schedule, and characterize Resources. To execute test cases using RETORCH, each test case must be annotated with at least one @AccessMode, and which refers to an already defined Resouce in the <SUT_NAME>SystemResources.json file. Additional information about the <SUT_NAME>SystemResources.json can be found in RETORCH Orchestration.

The tester needs to specify the access mode using the following attributes:

  • resID: Resource identifier for the access mode.
  • concurrency: The upper bound of test cases that can access the resource concurrently.
  • sharing: Allows sharing the resource between multiple test cases.
  • accessMode: The type of access mode performed by the test case.
@AccessMode(resID = "LoginService", concurrency = 10, sharing = true, accessMode = "READONLY")

The following code snippets illustrate a test case annotated with multiple @AccessMode annotations each one corresponding to a different Resource:

@AccessMode(resID = "LoginService", concurrency = 10, sharing = true, accessMode = "READONLY")
@AccessMode(resID = "OpenVidu", concurrency = 10, sharing = true, accessMode = "NOACCESS")
@AccessMode(resID = "Course", concurrency = 10, sharing = true, accessMode = "READONLY")
@ParameterizedTest
@MethodSource("data")
void forumLoadEntriesTest(String usermail, String password, String role) {
  this.user = setupBrowser("chrome", TJOB_NAME + "_" + TEST_NAME, usermail, WAIT_SECONDS);
  driver = user.getDriver();
  this.slowLogin(user, usermail, password);
}

RETORCH Orchestration

The RETORCH framework provides a generator that creates the Execution Plan, along with the required pipelining and script files for execution in a CI environment. The generation of scripts and pipelining code is based on the Access Modes annotated within the test cases and the Resource information specified in .retorch/configurations/<SUT_NAME>SystemResources.json.

The RETORCH orchestration generator requires 4 inputs:

  • The annotated E2E test cases with the RETORCH access modes into a single module Maven project.
  • A file with the Resources in JSON format(<SUT_NAME>SystemResources.json).
  • A properties file (retorchCI.properties) with the Environment configuration.
  • A custom docker-compose.yml file.

Given these inputs, the generator gives as output the necessary scripting code and the Jenkinsfile to execute the E2E test suite into a Continuous Integration system.

Prepare the E2E Test suite

The first step is to create several folders to store the configurations and place the docker-compose.yml in the single module project root. The resulting directory tree might look like as:

.

├── 📁 .retorch/
│   ├── 📁 configurations/
│   └── 📁 customscriptscode/
├── 📦 src
├── 🐳 docker-compose.yml
  • The 📁 .retorch/ directory would contain all the configuration files and scripting snippets that would be used to generate the pipelining code and the scripts to set up, deploy, and tear down the different Resources and TJob. Contains two subdirectories:
    • 📁 configurations/: stores the Resources and CI configuration files.
    • 📁 customscriptscode/: stores the different script snippets for the tear down, set up and environment.
  • The docker-compose.yml in the root of the project.
  • The different project directories and files.

The following subsections explain how to create each configuration file and how to prepare the docker-compose.ymlfile.

Create the Resource JSON file

The Resource file must be placed in the .retorch/configurations/ and named with the system or test suite name, followed by SystemResources.json (<SUT_NAME>SystemResources.json). This file contains a map with a series of Resources, using their unique ResourceID as a key. For each Resource the tester needs to specify the following attributes:

  • resourceID: A unique identifier for the Resource.
  • replaceable: A list of Resources that can replace the current one.
  • hierarchyParent: A resourceID of the hierarchical parent of the Resource.
  • elasticityModel: The elasticity model of the Resource, is composed by the following attributes:
    • elasticityID: A unique identifier for the elasticity model.
    • elasticity: Integer with the available Resources.
    • elasticityCost: Instantiation cost of each Resource.
  • resourceType: String with the type of the Resource(e.g. LOGICAL, PHYSICAL or COMPUTATIONAL).
  • minimalCapacities: List with the Minimal Capacities required by the Resource; each Capacity is composed by:
    • name: String between "memory", "processor" and "storage".
    • quantity: float with the amount of Capacity Required.
  • dockerImage: String with the concatenation of the placeholder name in the docker-compose, using ; as separator between placeholder and the image name.

The following snippet shows an example of two Resources declared in the JSON file:

{
  "userservice": {
    "hierarchyParent": [
      "mysql"
    ],
    "replaceable": [],
    "elasticityModel": {
      "elasticityID": "elasmodeluserservice",
      "elasticity": 5,
      "elasticityCost": 30.0
    },
    "resourceType": "LOGICAL",
    "resourceID": "userservice",
    "minimalCapacities": [
      {
        "name": "memory",
        "quantity": 0.2929
      },
      {
        "name": "processor",
        "quantity": 0.2
      },
      {
        "name": "storage",
        "quantity": 0.5
      }
    ],
    "dockerImage": "userservice;wigo4it/identityserver4:latest"
  },
  "frontend": {
    "hierarchyParent": [],
    "replaceable": [],
    "elasticityModel": {
      "elasticityID": "elasmodelfrontend",
      "elasticity": 1,
      "elasticityCost": 300.0
    },
    "resourceType": "LOGICAL",
    "resourceID": "frontend",
    "minimalCapacities": [
      {
        "name": "memory",
        "quantity": 2
      },
      {
        "name": "processor",
        "quantity": 1
      },
      {
        "name": "storage",
        "quantity": 0.88
      }
    ],
    "dockerImage": "frontend;nginx:latest"
  }
}

Create the retorchCI.properties file

The CI file must be placed in .retorch/configurations/, namely retorchCI.properties containing several parameters related to the SUT and the Continuous Integration Infrastructure, these parameters are the following:

  • agentCIName: the specific Jenkins agent used to execute the test suite.
  • sut-wait-html: state in the frontend (HTML displayed) when the SUT is ready to execute the test suite.
  • sut-location: location of the docker-compose.yml file used to deploy the SUT.
  • app-url: the URL where the frontend of the web application is made available.
  • testsBasePath: Path to the Java project root.

The following snippet provides an example of how this file looks like:

agentCIName=any
sut-wait-html=<title>Hello World</title>
sut-location=$WORKSPACE
app-url=https://full-teaching-$TJOB_NAME:5000
testsBasePath=./

Preparing the docker-compose.yml file

The orchestration generator also requires to parametrize the docker-compose.yml used to deploy the application by means including the necessary environment variables in the containers names and URIs, as well as the placeholders of the images specified above. Examples of the necessary changes in the docker-compose.yml can consulted in the FullTeaching and eShopOnContainers repositories:

(Optional) Specify script snippets to include in the set-up tear-down and environment

The RETORCH orchestration generator allows to specify scripting code/commands to be included in the generated set up, tear down, and the environment declaration of each TJob. To include it, the tester must create the following files in retorch/customscriptscode:

  • custom-tjob-setup: Contains the custom set up code (e.g. declare some environment variable specific for each TJob)or custom logging systems.
  • custom-tjob-teardown: Contains the custom tear down code (e.g. save some generated outputs).
  • custom.env: Contains configurations and environment variables common to all TJobs.
  • custom-coi-setup: Contains the custom set up code for the cloud or on-premise infrastructure (e.g. clone a repository and made it available for all TJobs).
  • custom-coi-teardown: Contains the custom tear-down code for the cloud or on-premise infrastructure (e.g. retrieve and calculate the coverage data or clean certain aspects of the environment).

Examples of the three snippets files can be consulted in FullTeaching Test Suite and eShopOnContainers.

Once created the different properties and configuration files, the single module directory tree might look like:

.
├── 📁 .retorch/
│   ├── 📁 configurations/
│   │   ├── {} <SUT_NAME>SystemResource.json
│   │   └── ⚙️ retorchCI.properties
│   ├── 📁 customscriptscode/
│   │   ├── 📄 custom-tjob-setup
│   │   ├── 📄 custom-tjob-teardown
│   │   ├── 📄 custom-coi-setup
│   │   ├── 📄 custom-coi-teardown
│   │   └── 🔐 custom.env
│   └── 📁 infra/
│       └── {} <SUT_NAME>CloudObjectInstances.json
├── 📦 src
├── 🐳 docker-compose.yml

Executing the Orchestration generator

Once all the files are created and the docker-compose.yml is prepared, follow these steps from the root of the SUT project (the same directory that contains .retorch/).

Step 1 — Compile the test classes and copy dependencies

The generator loads the annotated test classes at runtime using the Java ClassLoader, so they must be compiled and their transitive dependencies must be available before running the JAR. Run the following Maven command from the project root:

mvn test-compile dependency:copy-dependencies -DincludeScope=test

This compiles the test sources into target/ and copies all test-scoped dependency JARs into target/dependency/ (or the equivalent subdirectory if your project overrides outputDirectory).

Step 2 — Run the standalone JAR

java -jar retorch-orchestration-<version>-standalone.jar <rootPackageNameTests> <systemName> <jenkinsFilePath>

The JAR is available in Maven Central with the standalone classifier. The three arguments are:

  • rootPackageNameTests: root package where the annotated E2E test classes are located (e.g. com.sutexample.functional.tests).
  • systemName: system name that must match the name used in the Resources JSON file (e.g. sutexample).
  • jenkinsFilePath: directory path where the Jenkinsfile will be written (e.g. ./ for the project root).

For example, for the FullTeaching test suite:

mvn test-compile dependency:copy-dependencies -DincludeScope=test
java -jar retorch-orchestration-<version>-standalone.jar \
  com.fullteaching.e2e.no_elastest.functional.test \
  FullTeaching \
  ./

RETORCH Orchestration generator outputs

The generator provides four different outputs: the pipelining code, the necessary scripts to set up, tear down and execute the TJobs(.retorch/scripts/tjoblifecycles), the infrastructure(.retorch/scripts/coilifecycles) and the different environment files of each TJob ( .retorch/envfiles) :

  • ⛓️ Jenkinsfile: located in the root of the project, contains the pipelining code with the different stages in sequential-parallel that perform the different TJob lifecycle stages.
  • 📁 .retorch/scripts/tjoblifecycles and .retorch/scripts/coilifecycles contains the set up, execution, and tear down scripts for the TJobs and infrastructure
  • 📁 .retorch/envfiles: contains the generated custom environment of each TJob.

RETORCH Usage Profiler

The RETORCH framework provides a tool that generates the Usage Profiles for a given On-premise or Cloud Infrastructure. Given the execution data produced by each CI run and saved as artifact, the Execution Plan from the orchestration tool, and a Cloud Object Instance configuration file, the Usage Profiler is able to compute how the Contracted Capacities are used over time and render them graphically.

The RETORCH Usage Profiler requires the following inputs:

  • The execution data CSV file generated by the Jenkinsfile scripts (stored in the artifacts folder after each run).
  • The ExecutionPlan produced by the orchestration generator.
  • A {} <SUT_NAME>CloudObjectInstances.json file in 📁 .retorch/infra/ which describes the different deployment alternatives with their capacities, billing model, and lifecycle times.

To use the RETORCH Usage Profiler, we need to instantiate the UsageProfilerToolBox Add the retorch-profiling dependency to pom.xml:

<dependency>
  <groupId>io.github.giis-uniovi</groupId>
  <artifactId>retorch-profiling</artifactId>
  <version><!--SET HERE THE DESIRED VERSION--></version>
</dependency>

Create the Cloud Object Instances configuration file

The Cloud Object Instances file must be placed in .retorch/infra/ and named <SUT_NAME>CloudObjectInstances.json. It is a JSON array where each element describes one Cloud Object Instance (one infrastructure deployment alternative). Multiple entries allow comparing usage profiles across different providers or instance types.

For each Cloud Object Instance the tester must specify:

  • objectName: A unique identifier for the Cloud Object Instance.
  • billingOption: The billing model applied by the cloud provider, with:
    • billingName: Name of the billing plan (e.g. "As-you-go").
    • provider: Cloud provider name (e.g. "Azure").
    • invoicedPrices: A map of capacity name to price per unit (e.g. { "memory": 0.5, "slots": 1.20 }).
    • timePeriod: Minimum billing period in seconds (e.g. 3600 for hourly, 1 for per-second).
  • capacitiesContracted: A map of capacity name to a ContractedCapacity object with:
    • capacityName: One of memory, processor, storage, slots.
    • quantity: Total contracted amount.
    • granularity: Smallest provisionable unit (e.g. 32 for a full VM, 0.1 for containers, 1 for slots).

The COI lifecycle times (setup, execution, and teardown start/end) are derived automatically from the average lifecycle duration CSV produced in the next step — they do not need to be specified in the JSON file.

The following snippet shows an example with one VM-based Cloud Object Instance:

[
  {
    "objectName": "AzureVM",
    "billingOption": {
      "billingName": "As-you-go",
      "provider": "Azure",
      "invoicedPrices": {
        "memory": 0.5,
        "processor": 0.5,
        "storage": 0.5,
        "slots": 1.20
      },
      "timePeriod": 3600
    },
    "capacitiesContracted": {
      "memory": {
        "capacityName": "memory",
        "quantity": 32.0,
        "granularity": 32.0
      },
      "processor": {
        "capacityName": "processor",
        "quantity": 12.0,
        "granularity": 12.0
      },
      "storage": {
        "capacityName": "storage",
        "quantity": 32.0,
        "granularity": 32.0
      },
      "slots": {
        "capacityName": "slots",
        "quantity": 8.0,
        "granularity": 1.0
      }
    }
  }
]

An example of this file for the FullTeaching test suite can be found at FullTeachingCloudObjectInstances.json.

Generate the average lifecycle duration file from the CI execution data

Each execution of the RETORCH Execution Plan produces a CSV file stored in the artifacts folder. Collect one or more of these files into an executiondata folder; the tool computes the average durations across all runs.

Instantiate UsageProfilerToolBox and call generateAverageDurationCSVFile(), specifying the folder containing the execution data CSVs (inputPath) and the output file path (outputPath):

UsageProfilerToolBox usageProfiler = new UsageProfilerToolBox();
usageProfiler.

generateAverageDurationCSVFile("./executiondata","./averagedurationfile.csv");

Generate the raw TJob capacity-usage profile

Using the average duration file and the ExecutionPlan, generate a raw capacity-usage profile CSV that records how much of each capacity each TJob consumes at every second of the profiling window:

ProfileGenerator profileGenerator = new ProfileGenerator();
profileGenerator.

generateExecutionPlanCapacitiesUsage(
        plan,                        // ExecutionPlan from OrchestrationGenerator
        "./averagedurationfile.csv", // average duration CSV from previous step
                "./output/profile.csv",      // output path for the raw profile
                3600,                        // profiling window in seconds
                1                            // number of execution plan repetitions
);

Generate the COI Usage Profile charts

Pass the system name (used to locate the JSON config), the raw profile CSV, an output folder, and the plan name to generateCOIUsageProfiles(). The method reads every Cloud Object Instance from .retorch/infra/<systemName>CloudObjectInstances.json, computes the contracted-capacity overlay for each, and saves one set of PNG charts per instance:

usageProfiler.generateCOIUsageProfiles(
        "FullTeaching",                  // system name — loads FullTeachingCloudObjectInstances.json
                "./output/profile.csv",          // raw TJob profile from previous step
                "./averagedurationfile.csv",     // avg duration CSV — used to derive COI lifecycle times
                "./output/",                     // output folder for CSV files and chart images
        plan.getName()                   // plan name used to label the charts
);

For each Cloud Object Instance defined in the JSON file the profiler writes:

  • <outputPath>profile_<objectName>.csv — the profile with contracted-capacity rows added.
  • <outputPath><planName>-<objectName>-<capacityName>.png — one PNG chart per capacity.

Full example

The following template, used in the same test class as the orchestration generator, shows all steps together:

package com.sutexample.functional; // TO-DO Adjust the package name

import giis.retorch.orchestration.classifier.EmptyInputException;
import giis.retorch.orchestration.generator.OrchestrationGenerator;
import giis.retorch.orchestration.model.ExecutionPlan;
import giis.retorch.orchestration.orchestrator.NoFinalActivitiesException;
import giis.retorch.orchestration.scheduler.NoTGroupsInTheSchedulerException;
import giis.retorch.orchestration.scheduler.NotValidSystemException;
import giis.retorch.profiling.main.UsageProfilerToolBox;
import giis.retorch.profiling.profilegeneration.ProfileGenerator;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;

import java.io.IOException;
import java.net.URISyntaxException;

@Disabled("Exclude to execute this class when pushing the SUT")
class RetorchGenerateJenkinfileTest {
  @Test
  void testGenerateJenkinsfile() throws NoFinalActivitiesException, NoTGroupsInTheSchedulerException,
          EmptyInputException, IOException, URISyntaxException, NotValidSystemException, ClassNotFoundException {

    // Generate the Jenkinsfile (orchestration step)
    OrchestrationGenerator orch = new OrchestrationGenerator();
    orch.generateJenkinsfile("com.sutexample.functional.tests", "sutexample", "./");

    // Step 1: average lifecycle durations from Jenkins execution CSV files
    UsageProfilerToolBox usageProfiler = new UsageProfilerToolBox();
    usageProfiler.generateAverageDurationCSVFile("executiondata", "./averagedurationfile.csv");

    // Step 2: raw TJob capacity-usage profile over a 3600s window (1 execution)
    ExecutionPlan plan = orch.getExecutionPlan("com.sutexample.functional.tests", "sutexample");
    ProfileGenerator profileGenerator = new ProfileGenerator();
    profileGenerator.generateExecutionPlanCapacitiesUsage(plan, "./averagedurationfile.csv",
            "./output/profile.csv", 3600, 1);

    // Step 3: overlay contracted capacities and generate charts for each configured COI
    usageProfiler.generateCOIUsageProfiles("sutexample", "./output/profile.csv",
            "./averagedurationfile.csv", "./output/", plan.getName());
  }
}

Contributing

See the general contribution policies and guidelines for giis-uniovi at CONTRIBUTING.md.

Contact

Cristian Augusto - augustocristian@uniovi.es - Software Engineering Research Group (GIIS) - University of Oviedo, ES

Citing this work

RETORCH E2E Test Orchestration framework:

RETORCH*: A Cost and Resource aware Model for E2E Testing in the Cloud:

Acknowledgments

This work has been developed under the TestBUS (PID2019-105455GB-C32) and project supported by the Ministry of Science and Innovation (SPAIN)

Footnotes

  1. Henceforth, we will use the term "Resources" (capitalized) when referring to the ones required by the E2E test suite. ↩

About

RETORCH: Resource-aware End-to-End Test Orchestration

Topics

e2e-testingsystem-testingtest-automationtesting

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

Generated from giis-uniovi/samples-giis-template