Testing EmbSec Labs¶

This guide covers the testing infrastructure for EmbSec Kit, including unit tests, integration tests, and QEMU-based testing.

Overview¶

The EmbSec testing framework provides:

Unit Testing - Python-based tests for each lab
QEMU Integration - Automated testing in emulated environment
Test Framework - Reusable base classes for common patterns
CI/CD Integration - Automated testing in GitLab pipelines

Quick Start¶

# Run all lab tests
make test

# Test specific lab
make unittest-lab LAB=01-buffer-overflow

# Run tests with verbose output
python3 tools/scripts/test_labs.py -v

Test Framework Architecture¶

Base Classes¶

The framework provides base classes for common vulnerability patterns:

from test_framework import LabTestBase, BufferOverflowTestBase, FormatStringTestBase

# For buffer overflow labs
class TestLab(BufferOverflowTestBase):
    LAB_NAME = "01-buffer-overflow"
    BUFFER_SIZE = 64

    def get_exploit_payload(self, target, offset):
        # Implementation specific to lab
        pass

Test Structure¶

Each lab includes a tests/test_lab.py file that:

Verifies binary exists
Tests normal execution
Confirms vulnerability exists
Validates exploit works
Checks flag format and determinism

Writing Lab Tests¶

Basic Test Template¶

#!/usr/bin/env python3
import sys
import os
sys.path.append(os.path.join(os.path.dirname(__file__), '../../common'))

from test_framework import LabTestBase, p32

class TestLab(LabTestBase):
    LAB_NAME = "my-lab"
    EXPECTED_MENU_OPTIONS = [
        "1. Login",
        "2. Debug Info",
        "3. Exit"
    ]

    def test_04_vulnerability_exists(self):
        """Test that vulnerability is present"""
        self.start_qemu()
        # Lab-specific vulnerability test

    def prepare_exploit(self):
        """Gather information needed for exploit"""
        output = self.get_menu_choice("2")  # Debug info
        return {
            'target': self.extract_address(output, r'Target:\s*0x([0-9a-fA-F]+)')
        }

    def get_exploit_payload(self, **kwargs):
        """Generate the exploit payload"""
        target = kwargs['target']
        return b"A" * 64 + p32(target)

    def send_exploit(self, payload):
        """Send exploit to the target"""
        self.get_menu_choice("1")  # Login
        self.send_input(payload)

if __name__ == '__main__':
    import unittest
    unittest.main()

Test Methods¶

Required test methods:

Method	Purpose	Implementation
`test_01_binary_exists`	Verify build artifacts	Inherited
`test_02_normal_execution`	Check normal behavior	Inherited
`test_03_flag_not_accessible_normally`	Ensure security	Inherited
`test_04_vulnerability_exists`	Confirm vulnerable	Lab-specific
`test_05_exploit_gets_flag`	Validate exploit	Uses helpers
`test_06_flag_deterministic`	Check consistency	Inherited

Helper Methods¶

The framework provides utilities:

# QEMU interaction
self.start_qemu()                    # Start emulator
self.send_input(data)               # Send bytes/string
self.read_output(size=4096)         # Read response
self.get_menu_choice("1")           # Navigate menu

# Data extraction
self.extract_address(output, pattern)  # Parse addresses
self.extract_flag(output)             # Find flag

# Binary packing
p32(0x12345678)  # Pack 32-bit little-endian
p16(0x1234)      # Pack 16-bit
p8(0x12)         # Pack 8-bit

Running Tests¶

Command Line Interface¶

The test runner (tools/scripts/test_labs.py) supports:

# Test all labs
python3 tools/scripts/test_labs.py

# Test specific lab
python3 tools/scripts/test_labs.py -l 01-buffer-overflow

# Test labs matching pattern
python3 tools/scripts/test_labs.py -p buffer

# Verbose output
python3 tools/scripts/test_labs.py -v

# Custom build directory
BUILD_DIR=build-debug python3 tools/scripts/test_labs.py

Make Targets¶

# Run all tests
make test

# Test all labs
make unittest-labs

# Test specific lab
make unittest-lab LAB=01-buffer-overflow

# Test labs in QEMU (alias)
make test-labs

Test Output¶

Normal output:

Running tests for 2 labs...

Testing 01-buffer-overflow...
✓ 01-buffer-overflow - All tests passed (3.42s)

Testing 02-format-string...
✓ 02-format-string - All tests passed (2.89s)

============================================================
Test Summary
============================================================

Labs tested: 2
Passed: 2
Failed: 0
Skipped: 0

Total tests run: 12
Total failures: 0
Total errors: 0
Total time: 6.31s

Detailed report saved to test_report.json

Verbose output includes:

Individual test results
QEMU stdout/stderr
Detailed error messages

QEMU Testing Environment¶

Test Configuration¶

Tests run in QEMU with:

Board: LM3S6965EVB (Stellaris)
CPU: Cortex-M3
Serial: stdio
Monitor: disabled

Timeouts¶

Default timeouts:

Per-test timeout: 30 seconds
Read timeout: 2 seconds
QEMU startup: 1 second

Customize in test class:

class TestLab(LabTestBase):
    TIMEOUT = 60  # seconds

Debugging Failed Tests¶

Enable verbose mode to see QEMU output:

python3 tools/scripts/test_labs.py -v -l failing-lab

Or run QEMU manually:

qemu-system-arm -M lm3s6965evb -kernel build-qemu/labs/01-buffer-overflow/01-buffer-overflow -nographic -serial stdio -monitor none

Continuous Integration¶

GitLab CI Configuration¶

Tests run automatically in CI:

test:labs:
  stage: test
  script:
    - cmake -B build-qemu --toolchain sdk/cmake/LM3S6965Toolchain.cmake
    - cmake --build build-qemu --target labs
    - python3 tools/scripts/test_labs.py
  artifacts:
    when: always
    reports:
      junit: test_report.xml
    paths:
      - test_report.json

Test Reports¶

The test runner generates:

test_report.json - Detailed results
Console output with color coding
JUnit XML (when configured)

Testing Best Practices¶

1. Test Independence¶

Each test should be independent:

def setUp(self):
    """Fresh state for each test"""
    self.proc = None

def tearDown(self):
    """Clean up after each test"""
    if self.proc:
        self.proc.terminate()

2. Deterministic Exploits¶

Ensure exploits work consistently:

def test_06_flag_deterministic(self):
    """Run exploit multiple times"""
    flags = []
    for _ in range(2):
        # Get flag
        flags.append(flag)
    self.assertEqual(flags[0], flags[1])

3. Clear Assertions¶

Use descriptive assertions:

self.assertIsNotNone(
    target_addr, 
    "Could not find target address in debug output"
)

self.assertIn(
    "Access Granted",
    output,
    "Exploit did not bypass authentication"
)

4. Proper Timing¶

Allow time for operations:

self.send_input(payload)
time.sleep(0.5)  # Let exploit execute
output = self.read_output()

Advanced Testing¶

Custom Test Runners¶

Create specialized test scenarios:

class AdvancedTest(LabTestBase):
    def test_aslr_bypass(self):
        """Test ASLR bypass techniques"""
        for i in range(5):
            self.start_qemu()
            # Verify addresses change
            self.tearDown()
            self.setUp()

Performance Testing¶

Measure exploit reliability:

def test_exploit_reliability(self):
    successes = 0
    attempts = 10

    for _ in range(attempts):
        try:
            # Run exploit
            if flag:
                successes += 1
        except:
            pass

    reliability = successes / attempts
    self.assertGreater(reliability, 0.9, 
                      f"Exploit only {reliability*100}% reliable")

Fuzzing Support¶

Basic fuzzing integration:

def test_fuzzing(self):
    """Basic fuzzing test"""
    import random

    for _ in range(100):
        size = random.randint(1, 1000)
        data = os.urandom(size)

        self.start_qemu()
        self.send_input(data)
        # Check for crashes
        time.sleep(0.1)

Troubleshooting¶

Common Issues¶

Tests fail with "Lab binary not found"

# Ensure labs are built for QEMU
make qemu-build

"No module named test_framework"

# Run from project root
cd /path/to/embsec-kit
python3 labs/01-buffer-overflow/tests/test_lab.py

Timeout errors

Increase timeout in test class
Check if QEMU is hanging
Verify exploit completes quickly

Non-deterministic failures

Add delays after sending input
Check for race conditions
Ensure clean state between tests

Debug Mode¶

Run single test with debugging:

if __name__ == '__main__':
    # Run specific test
    import unittest
    suite = unittest.TestLoader().loadTestsFromName(
        'test_lab.TestLab.test_05_exploit_gets_flag'
    )
    unittest.TextTestRunner(verbosity=2).run(suite)

Test Coverage¶

Current Test Coverage¶

Each lab test covers:

Binary generation
Normal operation
Security properties
Vulnerability presence
Exploit functionality
Flag extraction

Adding New Tests¶

Extend existing tests:

class TestLab(BufferOverflowTestBase):
    def test_custom_check(self):
        """Additional lab-specific test"""
        self.start_qemu()
        # Custom verification

Integration with Development¶

Pre-commit Testing¶

Add git hook (.git/hooks/pre-commit):

#!/bin/bash
make test-labs || exit 1

VSCode Integration¶

Configure tasks (.vscode/tasks.json):

{
    "label": "Test Current Lab",
    "type": "shell",
    "command": "make unittest-lab LAB=${fileBasenameNoExtension}"
}

Next Steps¶

Debugging - Debug failed tests with GDB
Building - Build labs for testing
Environment Setup - Configure test environment