What is Apple ARKit?
ARKit is Apple’s framework for augmented reality apps and games. These turn your living room into the environment, or world, of the app. It’s part of iOS 11, but it’s going to be significantly upgrading in the upcoming iOS 12.
And it’s not just for the iPhone 8 family.
ARKit is a developer tool, it’s not something we iPhone users have to worry about too much. However, a look into what it’s about can tell us a lot about what the future looks like for augmented reality on the iPhone and iPad.
The software has been around for a little while now, so Apple is preparing to launch a new version, ARKit 2.0. We heard much more about the update at this year’s WWDC 2018, and we’ve updated the page below with the new developments.
Related: WWDC 2018
How does it work?
ARKit uses the iPhone’s cameras to map out an environment. A key part is recognising where the walls and floor are, establishing the basic geometry of the space.
It’s the Apple alternative to Google Project Tango, but puts fewer demands on the hardware. Project Tango requires a special camera array with dual cameras and an IR transmitter. This lets it make a rough 3D model of a room on the fly, and track motion with almost unnerving accuracy.
ARKit doesn’t require this. Instead, it recognises planes, like your floor. Then it uses the phone’s camera and its motion detectors to track movement as the iPhone is moved and tilted.
Motion co-processors like the iPhone 7 Plus’s M10 can efficiently track data from the gyroscope, accelerometer and magnetometer. This lets it monitor movement in a 3D space without excessive hardware demands. Google has actually announced something similar too, called ARCore. It’s Project Tango for phones without the flashy camera tech
The core of ARKit is about letting you drop objects into an environment, and manipulate them using the touchscreen. Other than recognising real-world objects, the main aim is to keep track of objects as the iPhone is moved. As soon as it loses track of this, the illusion is shattered.
One of the most obvious uses for this is interior design. You could place a chair in the corner of the room, and see how it looks as you turn the phone for a different view. It is like a more advanced version of some online glasses retailers’ virtual try-on feature.
However, Apple has also shown off how the functionality is being used in education, with videos showing children getting more information about a piece of art hanging on the wall in front of you, or being taught about the environment by projecting a landscape on a table.
We tend to think of far-fetched, futuristic real-world uses when we start thinking about AR and VR. But some of the best applications of ARKit are actually fairly mundane.
What’s new in ARKit 2.0?
At WWDC 2018 Apple announced a host of improvements that are coming to version 2.0 of ARKit.
The first of these improvements is USDZ, a new file format developed by Apple and Disney’s Pixar for ARKit to make it easier to create and share augmented reality content. The new format will be supported natively by Adobe’s Creative Cloud suite of software to allow for compatibility with common programs like Photoshop.
USDZ is the new file format Apple has developed for ARKit
On stage one demonstration showed an electric guitar being imported from a shopping website and then placed in the real world to get an impression of how big it actually is.
Apple also announced a new app called ‘Measure’, which will make use of the updated ARKit standard to allow you to measure objects in the real world using your phone’s camera. A demonstration showed this functionality off by allowing someone to virtually measure the size of a suitcase.
However, most exciting was multiplayer augmented reality, which allows multiple people to view the same virtual objects at the same time through different devices.
One augmented reality app from LEGO will let up to four players play with their characters in the same augmented reality environment, all interacting with the world in different ways.
The app combines virtual objects with real-world LEGO sets
What apps use ARKit?
It’s early days for ARKit. However, there are already lots of apps showing off what it can do (if only as trailer videos). Here are a few of the best demos.
Pokémon Go is probably the biggest game that uses ARKit right now. It employs the software to virtually place Pokémon in the real world with far more accuracy than it was able to previously. That said, most people who play the game turn off the AR functionality to make it easier to catch the monsters.
The Machines is a multi-player RTS game that lets several people play in the same AR environment. You control a troop of soldiers, in the vein of Command & Conquer.
It features advanced elements like audio that gets louder as you get closer to the action, and close to console-grade graphics.
Skyguide is a very neat astronomy app that pastes constellations onto the sky. We’re not talking made-up ones either. Using the compass and your location, it can determine the actual star charts for your position.
ModiFace 3D lets you see how you’d look with a different hair colour. It comes across a more advanced take on what some Snapchat filters do.
This is an interesting demo because it’s only really concerned with object recognition and tracking in a real space, not the room mapping so many associate with AR.
ARZombie is a classic example of an AR game. It puts you in a zombie home invasion scenario, 3D-modelled undead breaking in and attacking.
You fend them off, and build virtual defences in doors and windows to keep them out.
Rainforest Garage is a delightfully silly AR demo. You see an outline on a wall, and a blowtorch on the floor. Pick up the blowtorch and you can use it to cut out the outlined part of the wall.
It disappears, opening up to show a cartoony rainforest, like a low-rent Narnia.
AR Tape Measure is one of the ARKit apps we can really imagine using. The name explains it all.
You tap and drag out a virtual tape measure, which shows you the distance between objects. A similar tool is also one of the key uses for Project Tango. I’d probably still get the tape measure out for home decoration, but it seems pretty accurate already.
How does ARKit measure distance with “one eye”?
Some Apple phones have the dual cameras needed for traditional mapping of spaces. The iPhone 7 Plus, iPhone 8 Plus and iPhone X do. However, ARKit works with phones other than these too (see below). So how does it work?
No magic is required. An iPhone’s motion sensors let it approximate with surprising accuracy how much the device is moved or tilted.
ARKit may not be able to judge the layout of a room with a single, perfectly still view of a scene. But it can instantly re-analyse that same scene when your hand movements cause the phone to tilt by, say, two degrees.
This communication between the motion sensors and the camera more-or-less completely eradicates the issue of having one camera. I said this wasn’t magic, but it’s pretty close, right?
ARKit can use this progressive analysis of a space to build as accurate a portrait as Project Tango. In theory.
What are the limitations?
As with any form of AR, ARKit is fairly processor intensive. This means it’ll drain the battery quickly, and the phone/iPad may get warm or hot.
ARKit is not designed to cope with very fast motion either. Apple goes as far to explain to developers ways that they should politely advise users to slow down on the Apple Developer website.
Can I use it in any room?
How much space you’ll need depends on the kind of app being used. The hair changer Modiface 3D won’t need any at all.
However, if you’re hoping to play a full ARKit game, it’s not going to work that well if your living space consists of a bedroom that only just fits in a double bed. As the environment mapping relies on identifying planes, like the floor, a cluttered and/or cramped space probably won’t do the trick.
Which phones support ARKit?
ARKit isn’t just for the new iPhone 8, iPhone 8 Plus and iPhone X. It’s designed for iPhones and iPad that use the A9, A10 and A11 processors.
That includes the following devices:
- iPhone SE
- iPhone 6S
- iPhone 6S Plus
- iPad (2017)
- iPhone 7
- iPhone 7 Plus
- iPhone 8
- iPhone 8 Plus
- iPhone X
Related: iPhone 9
Have one of these? You just need iOS to try it out, once ARKit apps start appearing on the App Store.
Editorial independence
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
Professional conduct
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
Editorial independence
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
Professional conduct
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
