Every smart home product is put through a rigorous set of tests to gauge its performance and compatibility with other products. The exact process is broken down below.
One of the challenges of testing smart home products, is ensuring they work together. To this end we maintain a consistent test environment with a set of permanently installed products, to assess compatibility.
Test environment
All products are tested in a real home environment, with each product used in-situ as it would be when purchased. That means that light switches are wired into place, alarm sensors are fixed to windows or doors, and so on. Our test environment has the following products permanently installed for compatibility testing:
- Nest Thermostat
- Nest Protect
- Nest Cam IQ Indoor
- Philips Hue light bulbs
- LightwaveRF light switches
- Amazon Alexa
- Google Home
- Apple HomePod
- Samsung SmartThings
Some products, such as smart thermostats get their own tests, which we’ve listed below. However, all smart home products undergo the same set of criteria.
What we test
App control and ease of use: we test every product for ease of use, documenting any issues with configuration and set up. For each product, we fully test the smartphone and web apps available, looking for the range of features available, reliability and ease of use.
Voice assistant compatibility: we test every smart home product with every voice assistant that it’s compatible with: Apple Siri, Amazon Alexa and Google Assistant. Where Apple HomeKit is supported, we also test with the Home app to test features.
IFTTT compatibility: where If This Then That (IFTTT) is supported, we test to see what the triggers ad actions can achieve, such as turning on our Hue lights automatically when motion is detected by a camera. The exact tests depend on the options available.
SmartThings integration: where possible, products are tested with Samsung SmartThings to see how they perform and how they can be automated. Where support isn’t directly built in, we’ll test using third-party integrations or by using generic device handlers. For example, many Zigbee light bulbs can be used with SmartThings even if they’re not designed to work with the system.
‘Works with Nest’: where a product has Works with Nest certification, we test the automated rules to see how they work. For example, Hue bulbs can be turned on automatically when a camera detects a person.
Product tests
Where possible, we have individual tests for specific smart home products, as listed below.
Thermostats: all smart thermostats are wired into an electric radiator, controlled by a custom-built relay. The radiator is used to heat an outside office. All thermostats are tested over a period of two weeks, testing how it learns to adjust to the room. For smart radiator valves, we test using a hot water radiator.
We test every smart thermostat’s scheduling and any automated features, such as geo-location, which turns the heating on and off automatically based on location.
Smart speakers: all smart speakers are tested for a couple of weeks, testing their compatibility and ability to operate our common smart home kit. We grade them based on the ability to understand commands, and the range of products supported.
Security cameras: are installed in typical positions indoors or out, depending on the type. We test each one’s motion detection quality, looking for those that can avoid false positives in particular. We take sample image shots during the day and at night, to give us a comparison between each product.
Smart lighting: products are installed into common light fittings, and we can test GU10, bayonet, E14 and E27 bulbs. Standalone lighting (LED strips and the like) are tested by installing them in-situ.
We test for the quality of light (can you use it to replace existing light bulbs?), the range of lighting options (dimming, temperature range and colour options), and any additional lighting features, such as the ability to synchronise lights with music.
Related: Which smart bulb to buy?
Smart alarms: we install window/door and motion sensors as directed by the manufacturer. We measure how well the sensors respond to activity, and how easily the alarm can be programmed, such as turning off certain sensors at night. Where pet-safe sensors are available, we test them with our pet cats to ensure that they don’t trigger by accident.
Scoring and verdict
After all the tests are complete, we score the product using the criteria outlined in the criteria mentioned here. We first check to see if the product’s performance matches the manufacturer’s claims, and that all the features work as expected and advertised.
Value is a consideration during scoring, too. If a competing product offers equivalent features or performance for less money then this will affect the score. Equally, if a device is only slightly more expensive but performs significantly better then we’ll score accordingly.
Editorial independence
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
Professional conduct
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
Editorial independence
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
Professional conduct
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
