CI/CD Pipeline Guide¶

This guide explains the GitLab CI/CD pipeline used in the EmbSec Kit project. Understanding the pipeline helps you ensure your contributions pass all automated checks.

Pipeline Overview¶

The EmbSec Kit uses a multi-stage GitLab CI/CD pipeline to ensure code quality, test coverage, and reliable releases.

Pipeline Stages¶

Build: Compile code and create Docker images
Test: Run unit tests, integration tests, and security scans
Package: Create release artifacts
Deploy: Deploy documentation and create releases

graph LR
    A[Push Code] --> B[Build Stage]
    B --> C[Test Stage]
    C --> D[Package Stage]
    D --> E[Deploy Stage]

    B --> B1[Docker Images]
    B --> B2[SDK Build]
    B --> B3[Labs Build]
    B --> B4[Docs Build]

    C --> C1[Unit Tests]
    C --> C2[Lab Tests]
    C --> C3[Security Scan]

    D --> D1[SDK Package]
    D --> D2[Labs Package]

    E --> E1[Pages Deploy]
    E --> E2[Release Deploy]

Pipeline Jobs¶

Build Stage¶

`build:docker-arm`¶

Purpose: Build ARM cross-compilation Docker image
When: On main branch, tags, or docker-* branches
Output: Docker image for ARM builds

`build:docker-dev`¶

Purpose: Build development environment Docker image
When: On main branch, tags, or docker-* branches
Output: Docker image for local development

`build:docker-ci`¶

Purpose: Build CI-specific Docker image
When: On main branch, tags, or docker-* branches
Output: Optimized Docker image for CI jobs

`build:sdk`¶

Purpose: Build the EmbSec SDK library
Dependencies: Optional dependency on build:docker-arm
Output: Compiled SDK artifacts
Cache: Build directory cached by branch

`build:labs`¶

Purpose: Build all lab exercises
Dependencies: Requires build:sdk
Output: Compiled lab binaries and packages

`build:docs`¶

Purpose: Build project documentation
Tools: MkDocs with Material theme
Output: Static HTML documentation

Test Stage¶

`test:sdk`¶

Purpose: Run SDK unit tests
Tools: CTest
Timeout: 60 seconds per test
Requirements: Must pass for pipeline to continue

`test:labs`¶

Purpose: Test all labs in QEMU emulator
Tools: Custom test framework, QEMU
Output: JUnit test report
Coverage: All labs must pass their test suites

`test:security`¶

Purpose: Static security analysis
Tools:
cppcheck: Static analysis
scan-build: Clang static analyzer
Output: Security scan reports

Package Stage¶

`package:sdk`¶

Purpose: Create SDK release packages
Formats: .tar.gz and .zip
Naming: embsec-sdk-${COMMIT_SHA}.tar.gz

`package:labs`¶

Purpose: Package lab exercises
Format: Individual .zip files per lab
Output: Collected in single archive

Deploy Stage¶

`deploy:pages`¶

Purpose: Deploy documentation to GitLab Pages
When: Only on main branch
URL: Available at project's GitLab Pages URL

`deploy:release`¶

Purpose: Create GitLab releases
When: Only on tags
Includes: SDK and labs packages as release assets

Special Jobs¶

`nightly`¶

Purpose: Comprehensive nightly build and test
When: Scheduled (cron)
Configuration: Debug build with full testing

`audit:security`¶

Purpose: Deep security audit
When: Manual trigger only
Tools: cppcheck with aggressive settings, valgrind

Working with the Pipeline¶

Running Pipeline Locally¶

You can test pipeline jobs locally using Docker:

# Test SDK build
docker run --rm -v $(pwd):/workspace \
  gcc:11 \
  bash -c "cd /workspace && cmake --preset embedded && cmake --build build"

# Test with CI Docker image
docker-compose -f tools/docker/docker-compose.yml run --rm dev \
  make test

Understanding Pipeline Failures¶

Build Failures¶

Common causes:

Missing dependencies
Syntax errors
CMake configuration issues

Debugging:

# Check CMake configuration
cmake --preset embedded --trace

# Verbose build output
cmake --build build --verbose

Test Failures¶

Common causes:

Failing unit tests
Lab exploit issues
Timeout problems

Debugging:

# Run specific test
ctest -R test_name --output-on-failure

# Run with verbose output
python3 ./tools/scripts/test_labs.py -v

Security Scan Failures¶

Common causes:

Memory leaks
Buffer overflows
Uninitialized variables

Fixing:

# Run cppcheck locally
cppcheck --enable=all sdk/ labs/

# Run scan-build
scan-build cmake --build build

Pipeline Configuration¶

The pipeline is configured in .gitlab-ci.yml. Key configuration:

Variables¶

variables:
  DOCKER_IMAGE_BASE: "gcc:11"
  BUILD_DIR: "build"
  CMAKE_BUILD_TYPE: "Release"

Cache Configuration¶

cache:
  key: "${CI_COMMIT_REF_SLUG}"
  paths:
    - ${BUILD_DIR}/
    - .cache/

Job Dependencies¶

build:labs:
  needs: ["build:sdk"]  # Requires SDK to be built first

Optimizing for CI¶

1. Use Pipeline Cache¶

Cache expensive operations:

my_job:
  cache:
    key: "${CI_COMMIT_REF_SLUG}"
    paths:
      - vendor/
      - .cache/

2. Parallelize Tests¶

Split tests across jobs:

test:unit:
  parallel:
    matrix:
      - TEST_SUITE: [core, uart, gpio, timer]
  script:
    - ctest -R ${TEST_SUITE}

3. Fail Fast¶

Use when: on_failure for cleanup:

cleanup:
  stage: .post
  when: on_failure
  script:
    - echo "Cleaning up after failure"

4. Optimize Docker Images¶

Use multi-stage builds:

# Build stage
FROM gcc:11 as builder
RUN apt-get update && apt-get install -y cmake
COPY . /app
WORKDIR /app
RUN cmake --preset embedded && cmake --build build

# Runtime stage
FROM debian:bullseye-slim
COPY --from=builder /app/build/sdk /usr/local

Pipeline Best Practices¶

1. Keep Jobs Focused¶

Each job should have a single responsibility:

✅ test:uart - Test UART module
❌ test:everything - Test all modules

2. Use Artifacts Wisely¶

Only pass necessary files between stages:

artifacts:
  paths:
    - build/sdk/libembsec.a  # Specific files
  expire_in: 1 week          # Set expiration

3. Handle Flaky Tests¶

Retry flaky tests with limits:

test:integration:
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure

4. Secure Sensitive Data¶

Use GitLab CI/CD variables:

deploy:
  script:
    - echo "Deploying with key ${DEPLOY_KEY}"
  only:
    variables:
      - $DEPLOY_KEY  # Only run if variable exists

Monitoring Pipeline Performance¶

Pipeline Analytics¶

View in GitLab:

Project → Analytics → CI/CD Analytics
Track success rate, duration, and frequency

Job Timing¶

Add timing to scripts:

#!/usr/bin/env bash
start_time=$(date +%s)

# Your commands here

end_time=$(date +%s)
echo "Job took $((end_time - start_time)) seconds"

Resource Usage¶

Monitor Docker resource usage:

build:heavy:
  variables:
    DOCKER_MEMORY: "4g"
    DOCKER_CPUS: "2.0"

Troubleshooting¶

Common Issues¶

"No space left on device"¶

Cause: Build artifacts filling disk
Fix: Clean old artifacts, use expire_in

"Job exceeded timeout"¶

Cause: Test hanging or slow
Fix: Increase timeout or optimize test

"Cannot find package"¶

Cause: Missing dependency
Fix: Update Docker image or install in job

Debugging Techniques¶

1. Enable Debug Output¶

variables:
  CMAKE_VERBOSE_MAKEFILE: "ON"
  CI_DEBUG_TRACE: "true"

2. Add Debug Steps¶

script:
  - echo "Current directory: $(pwd)"
  - echo "Files: $(ls -la)"
  - echo "Environment: $(env | sort)"

3. Use Interactive Web Terminal¶

Available for debugging failed jobs in GitLab UI

4. Reproduce Locally¶

# Use same Docker image as CI
docker run -it --rm \
  -v $(pwd):/workspace \
  -w /workspace \
  registry.gitlab.com/embsec/kit/build-arm:latest \
  bash

# Run commands from failed job

CI/CD Variables¶

Predefined Variables¶

Commonly used GitLab CI variables:

CI_COMMIT_SHA: Git commit SHA
CI_COMMIT_REF_NAME: Branch or tag name
CI_PIPELINE_ID: Unique pipeline ID
CI_JOB_NAME: Current job name
CI_PROJECT_DIR: Project directory

Custom Variables¶

Define in .gitlab-ci.yml or project settings:

variables:
  BUILD_TYPE: "Release"
  ENABLE_COVERAGE: "true"

Variable Precedence¶

Trigger variables (highest)
Pipeline variables
Project variables
Group variables
Instance variables
.gitlab-ci.yml variables (lowest)

Contributing to CI/CD¶

Adding New Jobs¶

Define job in appropriate stage
Set proper dependencies
Configure caching if needed
Add meaningful artifacts
Document in this guide

Example:

test:new-module:
  stage: test
  needs: ["build:sdk"]
  script:
    - cd build
    - ctest -R new_module --output-on-failure
  artifacts:
    reports:
      junit: build/test-results-new-module.xml

Modifying Existing Jobs¶

Understand current behavior
Test changes locally
Consider impact on other jobs
Update documentation
Monitor after merge

Pipeline Maintenance¶

Regular tasks:

Update Docker images
Clean unused cache
Remove deprecated jobs
Optimize slow jobs
Update documentation

Resources¶

Getting Help¶

If you have CI/CD questions:

Check pipeline logs
Review this documentation
Search GitLab issues
Ask in merge request
Contact project maintainers

CI/CD Pipeline Guide¶

Pipeline Overview¶

Pipeline Stages¶

Pipeline Jobs¶

Build Stage¶

build:docker-arm¶

build:docker-dev¶

build:docker-ci¶

build:sdk¶

build:labs¶

build:docs¶

Test Stage¶

test:sdk¶

test:labs¶

test:security¶

Package Stage¶

package:sdk¶

package:labs¶

Deploy Stage¶

deploy:pages¶

deploy:release¶

Special Jobs¶

nightly¶

audit:security¶

Working with the Pipeline¶

Running Pipeline Locally¶

Understanding Pipeline Failures¶

Build Failures¶

Test Failures¶

Security Scan Failures¶

Pipeline Configuration¶

Variables¶

Cache Configuration¶

Job Dependencies¶

Optimizing for CI¶

1. Use Pipeline Cache¶

2. Parallelize Tests¶

3. Fail Fast¶

4. Optimize Docker Images¶

Pipeline Best Practices¶

1. Keep Jobs Focused¶

2. Use Artifacts Wisely¶

3. Handle Flaky Tests¶

4. Secure Sensitive Data¶

Monitoring Pipeline Performance¶

Pipeline Analytics¶

Job Timing¶

Resource Usage¶

Troubleshooting¶

Common Issues¶

"No space left on device"¶

"Job exceeded timeout"¶

"Cannot find package"¶

Debugging Techniques¶

1. Enable Debug Output¶

2. Add Debug Steps¶

3. Use Interactive Web Terminal¶

4. Reproduce Locally¶

CI/CD Variables¶

Predefined Variables¶

Custom Variables¶

Variable Precedence¶

Contributing to CI/CD¶

Adding New Jobs¶

Modifying Existing Jobs¶

Pipeline Maintenance¶

Resources¶

Getting Help¶

`build:docker-arm`¶

`build:docker-dev`¶

`build:docker-ci`¶

`build:sdk`¶

`build:labs`¶

`build:docs`¶

`test:sdk`¶

`test:labs`¶

`test:security`¶

`package:sdk`¶

`package:labs`¶

`deploy:pages`¶

`deploy:release`¶

`nightly`¶

`audit:security`¶