IOS Integration Testing by Square(corner.squareup.com)
corner.squareup.com
IOS Integration Testing by Square
http://corner.squareup.com/2011/07/ios-integration-testing.html
8 comments
Yup. I've setup a Jenkins server that uses a combination of xcodebuild and xcrun. The process:
- clones the repo from git
- builds with xcodebuild
- signs/provisions with xcrun
- creates a manifest file for over-the-air installs
- copies the build to a server where users can install
- saves off the symbol file for our crash server so it can symbolicate crashes
- sends emails
Thanks for the xcrun hint! Does that still handle the application/data deployment and execution, or does it just automate the packaging? All the information I can gather about doing OTA deployment points to still requiring the user to initiate the download, and then run it. I'd like to avoid that if possible because of the sheer number of libraries we need to test.
Have you tried xcodebuild? You should be able to execute unit tests from the command line with it.
scripting xcode UI to build a project? ouch, you're doing it wrong. Check out xcodebuild.
http://blog.jayway.com/2010/01/31/continuos-integration-for-...
http://blog.jayway.com/2010/01/31/continuos-integration-for-...
Any questions? We'll be around to answer them.
Does someone have to watch the integration test occur, or does it report errors to something?
It should report errors automatically. For the most part, it will detect problems automatically because a step will fail to execute. The reason is reported in the log that KIF creates. For example, if you had steps to
(1) enter the user name
(2) enter the password
(3) click "sign in"
(4) hit the refresh button
If the login failed for some reason, then step (4) would fail because it wouldn't be able to find the refresh button.
You can also add explicit verification steps, such as "wait for the view with the accessibility label 'Refresh'" if you want further validation.
If the login failed for some reason, then step (4) would fail because it wouldn't be able to find the refresh button.
You can also add explicit verification steps, such as "wait for the view with the accessibility label 'Refresh'" if you want further validation.
Thanks!
[deleted]
How hard would it be to add gesture support? double tap here, swipe left there, etc.
Are accessibility labels the only way to reference views?
Are accessibility labels the only way to reference views?
Also, keep in mind that many views just "do the right thing" as far as accessibility goes. Buttons for example will use their title as the default accessibility label if you don't set one explicitly.
Indeed, accessibility labels are the only way to reference views. Granted, your tests are running within your process, so they sky is pretty much the limit.
We're still considering the best way to implement gestures. There's some rudimentary support, but it's incomplete right now. Your input is welcome!
We're still considering the best way to implement gestures. There's some rudimentary support, but it's incomplete right now. Your input is welcome!
Nice! I've been looking for some proper iOS functional testing for a weekend project. Along with WaxSim this feels like enough plumbing to roll into a Jenkins plugin.
Why didn't you use UIAutomation? Was it lacking?
(http://developer.apple.com/library/ios/#documentation/Develo...)
(http://developer.apple.com/library/ios/#documentation/Develo...)
1. You can't automate running of the UIAutomation tests. Square really wanted this for continuous integration.
2. Writing the UIAutomation scripts is really tedious (though it's much better in iOS [redacted]).
3. The UIAutomation tests ran very slowly.
4. Having to write all the tests in JavaScript isn't very flexible.
That said, just like UIAutomation, KIF stands on the shoulders of the great accessibility features in iOS. In many ways KIF is like an Objective-C version of UIAutomation.
2. Writing the UIAutomation scripts is really tedious (though it's much better in iOS [redacted]).
3. The UIAutomation tests ran very slowly.
4. Having to write all the tests in JavaScript isn't very flexible.
That said, just like UIAutomation, KIF stands on the shoulders of the great accessibility features in iOS. In many ways KIF is like an Objective-C version of UIAutomation.
I’m kinda surprised by #2: I would think a scripting language to be ideal for this kind of thing.
Our iOS team at Square is more comfortable and efficient
writing tests (and test libraries) in Objective-C than JavaScript. We actually wouldn't gain much from the 'scripting language'-ness of JavaScript, and the additional language and tool (Instruments) would make writing tests more complicated and painful.
A nice collection of iOS testing posts:
http://softnoise.wordpress.com/2010/11/14/ios-setting-up-a-t...
http://softnoise.wordpress.com/2010/11/14/ios-setting-up-a-t...
how does this compare with selenium ?
Wow! I think I just came.
Seriously, unit testing in iOS is not great, and this kind of simulation is exactly what we need. Regular unit tests are great for model-layer stuff and data-processing code, but they will not help with most real-world problems that occur.
"Does my login flow still work?" and "Can I still use every feature of my app successfully?" is the kind of thing you need to know after committing code.
Seriously, unit testing in iOS is not great, and this kind of simulation is exactly what we need. Regular unit tests are great for model-layer stuff and data-processing code, but they will not help with most real-world problems that occur.
"Does my login flow still work?" and "Can I still use every feature of my app successfully?" is the kind of thing you need to know after committing code.
Thus far, my team has likewise been unable to find a good functional testing suite for iOS. We will definitely be giving this a shot.
Ok, this might be somewhat dumb; I'm just starting to get into testing. How does KIF[1] compare to Kiwi[2]?
--
[1]: http://corner.squareup.com/2011/07/ios-integration-testing.h...
[2]: http://http://www.kiwi-lib.info/
--
[1]: http://corner.squareup.com/2011/07/ios-integration-testing.h...
[2]: http://http://www.kiwi-lib.info/
Kiwi is for unit testing, KIF is for integration testing. They don't do the same thing (nor try to). Since you said you're just starting to get into it, here's what that means:
Integration tests simulate a user using your app - tapping on controls, manipulating views, typing in text, etc (see the video in the blog post). As these steps run you have simple tests to make sure the app is behaving as you expect it to.
Unit tests exercise more discrete things like model objects, controller logic, validations. Tons of small tests like "when we add these money values together, is the sum what we expect?".
At Square we use KIF for integration tests and GHUnit for unit tests. We run both on a continuous integration system (CruiseControl.rb) that runs the tests whenever we push new code.
Integration tests simulate a user using your app - tapping on controls, manipulating views, typing in text, etc (see the video in the blog post). As these steps run you have simple tests to make sure the app is behaving as you expect it to.
Unit tests exercise more discrete things like model objects, controller logic, validations. Tons of small tests like "when we add these money values together, is the sum what we expect?".
At Square we use KIF for integration tests and GHUnit for unit tests. We run both on a continuous integration system (CruiseControl.rb) that runs the tests whenever we push new code.
The best thing I got out of this is that I discovered Square's GitHub account... with an astonishing 65 repositories. I already held Square in high regard; this just icing on the cake.
Project followers might be a more relevant metric for impact than the number of repo's
Indeed. I have over a hundred repos, and I'm useless ;)
totally agreed :)
I discovered border_patrol, rack-timezone, rack-geo
At my current company we have a rather large automated test infrastructure that builds and runs ~50 or so individual libraries and their associated unit test programs on around 20 different configurations for 10 different platforms. Most of these platforms have a way of doing an unattended deployment of our test programs (and associated data) to the device, execute the programs, and retrieve the output.
This isn't working on iOS though because of its dependency to Xcode. We've tried scripting Xcode with AppleScript but this is fragile and routinely breaks.
Has anyone tackled this problem and succeeded?