Automated Memory Leak Testing on iOS

Did you know that you can automatically test for memory leaks when running UI tests for your iOS apps? You can. Here’s how.

Memory leaks are bad

Memory leaks can lead to app crashes, slowness and low energy efficiency, and more. Despite the fact that Swift language provides useful warnings (like for self in closure), sometimes the code can be more complicated, and it’s easy to overlook or miss leaks.

Common ways to discover leaks

The easiest, and most commonly-used tools are Instruments and Xcode’s Memory Graph Debugger.

Personally, I use Xcode Memory Graph Debugger when I get suspicious during development or in a code review. Instruments is great for a deeper dive into potential memory leaks.

Most leaks become visible pretty quickly, especially if you use a feature several times over and over again. For example, try opening and closing the screen five times and then open XCode > Debug Area > Debug Memory Graph. If you see instances related to destroyed screens - you have a leak.

Checking leaks takes time

The problem is that developers don’t always have the time to check leaks manually. It takes a while, is repetitive, and annoying. If only this could be done automatically….maybe every night on CI, for instance.

Automate that stuff!

We had an idea. Every night, somebody (e.g. Jenkins CI) could run and use our app (UI tests make good candidates). App usage would be recorded with Instruments, and we would extract newly-discovered memory leaks to compare against those we had previously discovered.

After experimenting a while with UI tests, Instruments and TraceUtility for reading .trace files we have connected all those tools in a shell script that can automate the detection of memory leaks.

It wasn’t all smooth sailing, however. When creating the script, we encountered several obstacles.

Obstacle #1: Simulated app usage

Most leaks pop up when using the app, and UI tests are (usually) good enough for simulating app usage. It’s not just that they guard the app’s visual/UX consistency, we can easily reuse them for discovering memory leaks.

Note that the example below uses Apple’s XCTestCase, whereas, at Showmax, we use a script with EarlGrey.

Obstacle #2: Attaching tested app to Instruments

Instruments can be launched via command line and set to export results into .trace files. While you can supply the path of application to be recorded, we did not find a reasonable way to use it for running UI tests (especially EarlGrey tests). So, we went with another option where the process ID of the running app must be supplied.

This needed a bit of juggling with Bash:

1. Start the test on the simulator using xcodebuild test-without-building
2. Wait for the app to start running on simulator
3. Capture its PID and pause its process
4. Start instruments -p <PID>
5. Allow some time for Instruments to prepare itself (otherwise it won’t record the beginning of test)
6. Resume app process in simulator
7. Now (finally), Instruments measures the app in the simulator
8. Results are exported into .trace files

Obstacle #3: Instruments needs a delay when the test is finished

We found it necessary to add a time delay after completing the test case, as Instruments needs some extra time to capture all data from the tested app. If you don’t give it that time, leaks might not be recorded at all. We added delays via NSPrincipalClass, where we specify the class that implements as XCTestObservation. Just pass the file path to the script and it adds code with testBundleDidFinish that will put the thread to sleep for a few seconds.

Obstacle #4: Instruments hangs when recording a long session

At first, we tried to record all tests together, but, in some instances, this caused Instruments to hang and it always produced large .trace files. So, we decided to let Instruments profile each test separately with xcodebuild -only-testing:<TestCaseName>.

Obstacle #5: How to read leaks from a .trace file

Unfortunately .trace files are big, full of raw data, and hard to read. Luckily, Qusic created TraceUtility. Qusic discovered undocumented frameworks that help to parse useful data (e.g. Time Profiler, Allocations, Network Connections) from .trace files. We forked the tool and added code that extracts memory leaks and saves them in plist. This enables us to give name to certain leak and count the occurrences.

Now, we store list of leak occurrences for every CI run. Then, in the next CI run, we can compare recently-found leaks with historical ones, and determine whether a new leak has appeared.

If so, an alert goes out to email, Slack, or Kibana, and we get moving.

Example

Here’s a little demonstration of how to automatically detect a simple memory leak using an app UI test.

protocol GeneratorDelegate: class {
    func didGenerate(number: Int)
}

class Generator {

    // Here is memory leak. To prevent, change to weak var.
    var delegate: GeneratorDelegate

    init(delegate: GeneratorDelegate) {
        self.delegate = delegate
    }

    func generate() {
        let number = Int.random(in: 0..<100)
        DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) {
            self.delegate.didGenerate(number: number)
        }
    }
}

The app has a screen that generates a random number. The screen references Generator class, which takes some time to generate a number, then returns a number via delegate. But the code contains a mistake - we forgot to define our var delegate as weak, resulting in a strong reference cycle.

Fortunately, we have a UI test that opens the generator screen and we can check it for leaks:

$ ./check_memoryleaks

Which results in:

Latest leaks:
 2x Malloc 32 Bytes
 1x UINavigationItem  ObjC  UIKitCore
 1x NumberGeneratorViewController  Swift  Leakmax  # <--- our leaking code
 1x CFDictionary  ObjC  CoreFoundation
 1x Malloc 48 Bytes
 1x NumberGenerator  Swift  Leakmax                # <--- our leaking code
 1x UITabBarItem  ObjC  UIKitCore

🎉 Hooray, we discovered leak!

Known limitations

• Only for simulators. Not tested with real devices. To measure with Instruments, we would need to figure out how to retrieve the process ID of the tested app on a real device. We would also have to pause unit test PID until Instruments starts, or figure out how to let Instruments start the unit test process itself.
• Only UI tests. Not tested with unit tests, but it should work similarly. It’s likely that, instead of pausing App Runner, we would need pause Simulator.

Conclusion

Automated testing for memory leaks has some constraints, but it’s both possible and useful. You can find our script on Github and try it yourself. What we’d really like is to hear about your experiences with memory leak testing. How do you check for memory leaks? Let us know.