Repository navigation
Run every test runner in the Android emulator tests - #327
niktishakov wants to merge 2 commits into
Conversation
With Swift Build, each test target has its own test runner. The script ran them with the glob ./*-test-runner, so the shell passed the other runners as arguments to the first one and only the first test target ran. In swift-foundation, FoundationInternationalizationTests and FoundationMacrosTests didn't run on the emulator. Run the test runners one at a time, and add a second test target to TestPackage so that the emulator tests cover more than one runner. Fixes swiftlang#277.
| TEST_SHELL="${TEST_SHELL} && ${TEST_CMD} --testing-library swift-testing && [ \$? -eq 0 ] || [ \$? -eq 69 ]" | ||
| for TEST_RUNNER in ${TEST_RUNNERS}; do | ||
| TEST_SHELL="${TEST_SHELL} && echo 'Running ${TEST_RUNNER}'" | ||
| TEST_SHELL="${TEST_SHELL} && ${TEST_RUNNER} --testing-library xctest" |
There was a problem hiding this comment.
Can we change these commands to be semi-colon-delimited instead? Otherwise, I think any failing test in one runner would stop the other runners from being executed, whereas we should always run all the runners.
|
Thanks, this is great, 😃 @marcprux, please review. @niktishakov, any Android pulls like this, feel free to ping me or the @swiftlang/android-workgroup. |
Separate the commands with `;` so that a failing runner doesn't stop the others, and keep a `failed` flag so the job still fails if any runner failed.
|
@rnro do we need something similar for the new unified workflows? |
|
Rick's pull invokes this same Android emulator script, so no changes will be needed there. |
|
Marc may be busy, @jakepetroules, could you take a look? |
|
@justice-adams-apple, please run the CI here. |
|
@shahmishal, not getting a response from other committers to this repo, mind running the workflows on this? If it's running well with the test, we can get this in. |
finagolfin
left a comment
finagolfin
left a comment
There was a problem hiding this comment.
Thanks, CI shows this working, with the three Windows failures unrelated.
Fixes #277.
With Swift Build, each test target gets its own
*-test-runner.android-emulator-tests.shran them with./*-test-runner --testing-library …, so the shell passed the other runners as arguments to the first one, and only the first test target ran.In swift-foundation, the staging directory has three runners, but only
FoundationEssentialsTestsran on the emulator.FoundationInternationalizationTests(485 tests, including Calendar and the ICU-based tests) andFoundationMacrosTests(45 tests) never ran on Android. This includes the formatting tests for the Swift implementations that replace ICU, like theListFormatStylesuite from swiftlang/swift-foundation#1991 and the date pattern tests in swiftlang/swift-foundation#2330.This runs the test runners one at a time, with the same
xctestandswift-testingpasses and the same handling of exit code 69 as before. A failure in one runner or pass doesn't stop the others: every runner runs both passes, and the job fails at the end if any of them failed. If there are no*-test-runnerfiles, it falls back to*.xctestas before, so the native build system still works. It also adds a second test target totests/TestPackage, so this repository's own Android check covers more than one runner.About going back to a single runner, as asked in #277: running each runner works with both build systems and doesn't need a change in SwiftPM.
Effect on other packages. The scripts are checked out from
main, so this applies right away to every package that usesenable_android_sdk_checks. In packages with more than one test target, test targets that didn't run before will run, so failures that were hidden can show up, and the Android job takes longer. I ran the same fork check on two of them:One more difference: exit code 69 is now accepted only from the
swift-testingpass, as the comment says. Before, because of how&&and||combine, it was also accepted from thexctestpass. In these runs, thexctestpass returns 0 when it finds no tests.Testing:
sh,bash --posixanddash: all runners run; exit code 69 fromswift-testingdoesn't fail the run; a failing runner or pass fails the run, and everything after it still runs.Target1Tests,Target2Testsstill ran and passed, and the job failed; without it, both runners passed and the job passed.swift testfortests/TestPackageruns both test targets.shellcheckreports nothing for the script.