Photo by JESHOOTS.COM on Unsplash
git bisect run xcodebuild
Automatically find a commit that introduced a bug in a large and frequently updated iOS application
This blog post explains how you can easily find a commit that introduced a bug in an iOS application.
Motivation
The following technique is helpful if you work with many developers on a large repository with daily complex commits. Once you find the commit, you can analyze the file changes to determine the root cause of the bug.
Technique
Git, as source-control management (SCM) system, provides a helpful command called git bisect
.
This command uses a binary search algorithm to find which commit in your projectโs history introduced a bug. You use it by first telling it a "bad" commit that is known to contain the bug, and a "good" commit that is known to be before the bug was introduced. Then git bisect picks a commit between those two endpoints and asks you whether the selected commit is "good" or "bad". It continues narrowing down the range until it finds the exact commit that introduced the change.
You could perform a manual test to check if it is a "good" or "bad" commit for every iteration, but this is very tedious. Instead, can you write a test or script to detect the bug? Often this is possible even if you don't know what code caused the problem.
If you have a script that can tell if the current source code is good or bad, you can bisect by issuing the command
$ git bisect run my_script arguments
You can use git bisect run xcodebuild test ...
once you enhanced an existing XCTestCase
to detect the bug through UI tests. Then Git can perform every iteration automatically and will spill out the information which was the first "bad" commit.
I recommend outsourcing the xcodebuild
command in a Makefile
for readability.
Example for a Makefile
:
uitest:
xcodebuild test \
-project FlakyApp.xcodeproj \
-scheme FlakyAppUITests \
-sdk iphonesimulator \
-destination 'platform=iOS Simulator,name=iPhone 13'
Now you can call git bisect run make uitest
:)
Another advantage of outsourcing is that you can tweak the xcodebuild
command and still stably trigger git bisect run
. For example, let's run only the one test to detect the bug by specifying -only-testing
option.
uitest:
xcodebuild test \
-project FlakyApp.xcodeproj \
-scheme FlakyAppUITests \
-sdk iphonesimulator \
-destination 'platform=iOS Simulator,name=iPhone 13' \
-only-testing "FlakyAppUITests/FlakyAppUITests/testExample"
This change will make the bisection execution faster and you still trigger it with git bisect run make uitest
.
Example
The following simplified repository contains an iOS application, and a bug was introduced at some point.
We only know the last "good" commit (2de359e
) and one "bad" commit for which the app does not work as expected (6e02ff9
).
In this example the bug is that the value of a SwiftUI Toggle is no longer true
. We can update the UI Test in FlakyAppUITests.swift
to check for that.
func testExample() throws {
let app = XCUIApplication()
app.launch()
let isValidSwitch = XCUIApplication().switches["isValidSwitch"]
XCTAssertTrue(isValidSwitch.value as? String == "1")
}
Now let's start bisecting the commits.
git bisect start 6e02ff9f1e319fe095a9dcfec092f76c3005355c 2de359e2ce524d9d4e8276595cc3c70ae9311996
Git tells us that roughly 2 steps are necessary to find the culprit.
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[7e68c4102f3d68310285fe58acbcf9d21503ff51] maybe
Then we tell Git to run make uitest
for every iteration.
git bisect run make uitest
Git starts processing ...
** TEST SUCCEEDED ** [9.950 sec]
Bisecting: 1 revision left to test after this (roughly 1 step)
[65c0d1ac4489241306692be4a7addbb38703aff3] maybe !
...
** TEST FAILED **
make: *** [uitest] Error 65
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[c8da8ee9fa48736b63bdbec0e1e1a179612396b3] maybe
...
65c0d1ac4489241306692be4a7addbb38703aff3 is the first bad commit
commit 65c0d1ac4489241306692be4a7addbb38703aff3
Author: Marco Eidinger <eidingermarco@gmail.com>
Date: Wed Apr 20 08:45:34 2022 +0200
maybe !
FlakyApp/ContentView.swift | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Voila, we know the first bad commit is 65c0d1a
!
Now that we know which commit is the culprit, we call the following commit to return to the original HEAD.
git bisect reset
Note: In this simplified example, we could have used git blame
to identify the commit & root cause because there are not so many commits and the commits are very small. But imagine there are fifty commits with each for more than ten file changes of hundred lines!
Summary
git bisect
is perfect when you don't know what code caused a problem and are working on a large and frequently updated project.- Leverage tests to automate
git bisect
.- For iOS app use
xcodebuild test
. - For a Swift Package you can use
xcodebuild
as well orswift test
.
- For iOS app use