Build-time test discovery + static native test codegen (experimental, opt-in)

[piotruela]

What & why

Stock Patrol discovers tests at runtime (dynamic registration once the app
launches), so no static list of test identifiers exists before a run. On iOS —
where XCUITest has no built-in sharding — that blocks per-test sharding entirely.

This PR adds experimental, opt-in build-time test discovery: a host
flutter test serializes the Dart test tree to a manifest at build time, and we
generate a real, statically-declared native test per Dart test. Every test becomes
individually addressable (-only-testing on iOS, am instrument -e class#method
on Android).

This should unblock iOS test sharding on device farms that shard by an explicit
test list — SauceLabs (testListFile) and BrowserStack — plus balanced Android
sharding and cleaner per-test report names.

What's included

• iOS static XCTest codegen (new STATIC runner macros) + Android static JUnit codegen.
• Config opt-in: patrol.emit_test_manifest: true (the --emit-test-manifest
  flag overrides it); iOS build fails fast if RunnerUITests.m isn't switched to
  the static macro.
• patrol test --no-build [--only "<dart name>"]: re-run built tests without
  rebuilding (iOS xcodebuild test-without-building, Android am instrument).
• Discovery reporting (tests grouped by source file) + docs page.

Usage

# pubspec.yaml
patrol:
  emit_test_manifest: true

patrol build ios --emit-test-manifest         # or: patrol build android
patrol test --no-build --only "my group my test"   # one test, no rebuild

iOS also needs the static macro in RunnerUITests.m (see docs; the CLI prints the
exact snippet if missing).

Verification

Unit tests (manifest/selector/codegen/config) + dart analyze clean. Verified
end-to-end in dev/e2e_app: iOS single test via -only-testing → Executed 1 test; Android via am instrument → OK (1 test); --no-build --only runs one
test with no rebuild on both platforms.

Limitations

Experimental. Host-side discovery can diverge for tests registered conditionally
on the runtime env (Platform.isX, dart-define-derived names). Tests must live
outside integration_test/ (use patrol_test/). Parallel simulators on one host
collide on Patrol's fixed ports — shard across separate hosts/VMs (as farms do).
Initial build is slightly slower (paid once; --no-build re-runs skip building).

[github-actions]

🚀 Preview Deployment Ready!

Preview URL: https://pr-3197-patrol-docs.vercel.app

---

Latest Deployment

• Commit: dfd13a1
• Branch: feat/ios-discovery-step
• Triggered by: @piotruela
• Deployed at: 2026-07-28 20:35:54 UTC

_{This preview URL is stable and will be updated with each new commit to this PR.}

[Copilot]

Pull request overview

Adds opt-in build-time Patrol test discovery, static native test generation, and no-build test reruns for Android and iOS.

Changes:

• Generates manifests and native XCTest/JUnit tests.
• Adds configuration, selection, and --no-build CLI support.
• Adds tests, documentation, and an end-to-end example.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
packages/patrol/lib/src/constants.dart	Adds discovery-mode flag.
packages/patrol/lib/src/common.dart	Registers no-op tests during discovery.
packages/patrol/darwin/patrol/Sources/PatrolImpl/PatrolTestManifest.swift	Reads iOS manifests.
packages/patrol/darwin/patrol/Sources/patrol/include/PatrolIntegrationTestIosRunner.h	Adds static XCTest runner macros.
packages/patrol/android/src/main/kotlin/pl/leancode/patrol/PatrolJUnitRunner.java	Supports explicit skip values.
packages/patrol_cli/test/general/pubspec_reader_test.dart	Tests manifest configuration.
packages/patrol_cli/test/crossplatform/test_manifest_test.dart	Tests parsing and selectors.
packages/patrol_cli/test/android/android_test_codegen_test.dart	Tests JUnit generation.
packages/patrol_cli/lib/src/test_bundler.dart	Serializes discovery manifests.
packages/patrol_cli/lib/src/runner/patrol_command.dart	Defines new CLI options.
packages/patrol_cli/lib/src/pubspec_reader.dart	Reads the opt-in setting.
packages/patrol_cli/lib/src/ios/xcode_test_codegen.dart	Generates XCTest methods.
packages/patrol_cli/lib/src/ios/ios_test_backend.dart	Integrates iOS discovery and selection.
packages/patrol_cli/lib/src/crossplatform/test_manifest.dart	Models manifests and native names.
packages/patrol_cli/lib/src/crossplatform/test_manifest_generator.dart	Runs host-side discovery.
packages/patrol_cli/lib/src/crossplatform/app_options.dart	Builds discovery and filtered-run commands.
packages/patrol_cli/lib/src/commands/test.dart	Adds no-build execution flow.
packages/patrol_cli/lib/src/commands/build_ios.dart	Adds iOS opt-in support.
packages/patrol_cli/lib/src/commands/build_android.dart	Adds Android opt-in support.
packages/patrol_cli/lib/src/android/android_test_codegen.dart	Generates static JUnit classes.
packages/patrol_cli/lib/src/android/android_test_backend.dart	Integrates Android generation and reruns.
docs/documentation/ci/meta.json	Registers the documentation page.
docs/documentation/ci/build-time-test-discovery.mdx	Documents the feature.
docs/cli-commands/test.mdx	Documents no-build testing.
docs/cli-commands/build.mdx	Documents build-time discovery.
dev/e2e_app/patrol_test/dynamic_test.dart	Adds dynamic discovery examples.
dev/e2e_app/ios/RunnerUITests/RunnerUITests.m	Enables the static runner.
dev/e2e_app/.gitignore	Ignores generated native tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

packages/patrol_cli/lib/src/android/android_test_codegen.dart:194

• Although _locateHostTest explicitly accepts .kt hosts, this regex only recognizes Java's FooActivity.class. A Kotlin host uses FooActivity::class.java, so a supported custom activity is silently replaced with MainActivity, potentially producing uncompilable or incorrect generated tests. Parse both forms.

  _HostActivity _parseActivity(String content) {
    final match = RegExp(
      r'setUp\(\s*([A-Za-z_][\w.]*)\.class',
    ).firstMatch(content);
    final token = match?.group(1) ?? 'MainActivity';

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (11)

packages/patrol_cli/lib/src/android/android_test_codegen.dart:224

• The host locator scans Kotlin files, but this expression only recognizes Java's FooActivity.class. A Kotlin host normally calls setUp(FooActivity::class.java), so a custom activity is silently replaced with MainActivity; the import parser also requires Java's semicolon. The generated test then launches or compiles against the wrong activity. Parse both Java and Kotlin forms.

    final match = RegExp(
      r'setUp\(\s*([A-Za-z_][\w.]*)\.class',
    ).firstMatch(content);

packages/patrol_cli/lib/src/android/android_test_backend.dart:683

• When no flavor is supplied, this accepts every APK with the requested build-mode segment, including stale flavored APKs under paths such as apk/dev/debug. Since listSync() ordering is unspecified, --no-build can install a stale flavor (or pair app and test APKs from different variants) instead of the unflavored artifacts. Resolve the exact standard app/test paths for the requested flavor and mode, as BuildAndroidCommand.printApkPaths already does, rather than taking the first recursive matches.

    bool matches(File apk) {
      final segments = apk.path.split(RegExp(r'[/\\]'));
      if (!segments.contains(buildMode)) {
        return false;
      }
      // When a flavor is set, its (case-sensitive) directory segment is present
      // in both the app and androidTest APK paths; use it to disambiguate.
      if (flavor != null && !segments.contains(flavor)) {
        return false;
      }
      return true;

packages/patrol_cli/lib/src/android/android_test_backend.dart:773

• This prefix match can select an unrelated installed application (for example, configured package com.foo also matches target com.foobar). If no exact instrumentation exists, --no-build may invoke the wrong runner. Restrict fallback matching to the dot-delimited applicationIdSuffix form.

    final prefixed = entries
        .where((e) => e.target != null && e.target!.startsWith(packageName))
        .toList();

packages/patrol_cli/lib/src/runner/patrol_command.dart:242

• addMultiOption splits comma-separated values by default, but Dart test names can legitimately contain commas. For example, --only "opens menu, then closes it" is parsed as two requested names and will never match the manifest. Disable comma splitting so each repeated --only occurrence remains an exact test name.

      ..addMultiOption(
        'only',
        help:
            'With --no-build, run only the test(s) whose exact Dart name '
            '(as printed during discovery) is given. Repeatable; omit to run '
            'all discovered tests.',
      );

packages/patrol_cli/lib/src/ios/ios_test_backend.dart:285

• The error is raised only when no requested name matches. With multiple --only values, one valid name plus one typo silently runs the valid test and reports success without running the other requested test. Track matched names and fail if any distinct requested name is absent.

    for (var i = 0; i < tests.length; i++) {
      if (onlyTests.contains(tests[i].dartName)) {
        out.add(selectors[i]);
      }
    }
    if (out.isEmpty) {
      throwToolExit(
        'None of the requested --only test(s) were found in the manifest.\n'
        'Available tests:\n${tests.map((t) => '  ${t.dartName}').join('\n')}',
      );

packages/patrol_cli/lib/src/android/android_test_backend.dart:647

• The error is raised only when no requested name matches. With multiple --only values, one valid name plus one typo silently runs the valid test and reports success without running the other requested test. Track matched names and fail if any distinct requested name is absent.

This issue also appears in the following locations of the same file:

• line 673
• line 771

    for (var i = 0; i < tests.length; i++) {
      if (onlyTests.contains(tests[i].dartName)) {
        out.add('$fqcn#${methods[i]}');
      }
    }
    if (out.isEmpty) {
      throwToolExit(
        'None of the requested --only test(s) were found in the manifest.\n'
        'Available tests:\n${tests.map((t) => '  ${t.dartName}').join('\n')}',
      );

packages/patrol_cli/lib/src/crossplatform/test_manifest_generator.dart:92

• An empty manifest is treated as a successful discovery, but Android codegen then emits a class with zero @Test methods and execution is restricted to that class. JUnit reports No runnable methods, so the build succeeds but the resulting test artifact is unusable. Reject an empty discovery before codegen, or generate an explicitly ignored placeholder test.

    final tests = manifest.tests;
    if (tests.isEmpty) {
      _logger.info('No Dart tests discovered.');
      return;

packages/patrol_cli/lib/src/crossplatform/app_options.dart:90

• The discovery-only defines are emitted before caller-provided dartDefines and --dart-define-from-file values. A project that already defines either reserved key can therefore override PATROL_TEST_DISCOVERY=true or redirect PATROL_MANIFEST_OUTPUT, causing discovery to run normally or report a missing manifest. Emit these two internal defines last so they are authoritative.

      ...['--dart-define', 'PATROL_TEST_DISCOVERY=true'],
      ...['--dart-define', 'PATROL_MANIFEST_OUTPUT=$manifestOutputPath'],
      for (final dartDefine in dartDefines.entries) ...[
        '--dart-define',
        '${dartDefine.key}=${dartDefine.value}',
      ],
      for (final dartDefineFromFilePath in dartDefineFromFilePaths) ...[
        '--dart-define-from-file',
        dartDefineFromFilePath,

packages/patrol_cli/lib/src/crossplatform/test_manifest_generator.dart:44

• This single global manifest is later used to map --only names to methods in already-built platform artifacts, but every iOS or Android discovery overwrites it. Building Android after iOS (or rebuilding with different targets/defines) can therefore make a later iOS --no-build --only select methods from the Android manifest, and vice versa. Non-manifest builds also leave this file stale. Persist/load a manifest keyed to the platform and concrete build artifact (or record and validate build identity) so selectors cannot be resolved against an unrelated binary.

    final manifestFile = _rootDirectory
        .childDirectory('build')
        .childDirectory('patrol')
        .childFile('patrol_test_manifest.json');

packages/patrol/lib/src/common.dart:117

• This early return bypasses the existing validation that rejects supplying both nativeAutomatorConfig and platformAutomatorConfig. Discovery therefore succeeds for a test bundle that will throw during normal on-device registration, producing static tests that cannot execute. Keep argument validation before the discovery branch so build-time and runtime registration enforce the same contract.

  if (constants.testDiscoveryEnabled) {
    // Build-time discovery (host `flutter test`): only register the test so it
    // shows up in the group tree. Do NOT initialize PatrolBinding (Live binding,
    // incompatible with the host automated binding) and do NOT run the body
    // (it would block on waitForExecutionRequest()).

packages/patrol_cli/lib/src/android/android_test_backend.dart:505

• This new execution path has no automated coverage, although AndroidTestBackend is unit-tested elsewhere. It contains artifact selection, manifest-to-method mapping, instrumentation parsing, process failure detection, and two APK installs; regressions here can silently run the wrong variant or report failed instrumentation as successful. Add backend tests for all/selected tests, missing artifacts, flavored APK resolution, custom instrumentation, and failing am instrument output.

  Future<void> executeWithoutBuilding(