AI Can Map Plants in 3D

University of Canterbury researchers have developed AI technology capable of seeing behind dense foliage to create detailed 3D models of plants, opening new possibilities for crop forecasting and horticultural automation

The technology uses artificial intelligence to identify, measure and track fruit that would otherwise be hidden by leaves. Led by Computer Science Professor Richard Green, alongside researcher Dr Richie Ellingham, the team uses cameras, lighting and a complex pipeline of AI systems to digitally remove foliage and reconstruct an accurate 3D model of a plant.

“You can’t automate what you can’t see. By creating a complete 3D model of each plant, including what is behind the leaves, we can open the door to automating almost any repetitive manual task associated with that plant,” said Green.

Seeing the Whole Plant

According to Green, agricultural environments present a particular challenge for automation because every tree, vine, branch and piece of fruit is different.

To overcome that variability, the technology is designed to model individual plants, including structures and fruit obscured by foliage.

“The system can identify individual apples, cherries or grapes, show which branch or cane they are attached to, and calculate measurements including their size, volume and surface area. Repeated scans can also track how much an individual piece of fruit has grown,” explained Green.

One potential application is improving the accuracy of pre-harvest crop forecasting.

Growers currently rely largely on people to sample, count and measure fruit, which Green says can result in inaccuracies of up to 23%. During testing, the technology has achieved fruit counts within about 2% to 3% of the correct figure.

“For large growers, better information could potentially save millions of dollars and reduce waste across their operations,” added Green.

Accurate forecasting could help growers determine workforce, packaging and storage requirements ahead of harvest.

Automation Opportunities

UC Vision has developed mobile camera systems that travel along vineyard and orchard rows. Its vineyard system uses two rows of cameras to capture images through dense vine canopies, while a taller structure can scan orchard trees reaching approximately 3.5m.

Professor Green says the resulting models could eventually guide autonomous machinery carrying out pruning, thinning, spraying and harvesting.

While the team’s current ground-based pruning work is focused on vineyards and orchards, Professor Green said its longer-term ambitions extend into landscaping.

The team is also working on aerial-based pruning of pine forests and branches near power lines.

Professor Green says its recent AI breakthrough in creating highly detailed 3D models of trees could ultimately enable “autonomous general landscaping pruning – both ground-based and aerial-based as appropriate”.

That could take technology currently being developed around highly structured horticultural tasks into the broader landscape maintenance environment.

The breakthrough builds on 15 or so years of UC research projects supported by more than $32m in government investment in computer vision, artificial intelligence and agricultural robotics.

The team has collected 3D modelling data across 20 commercial farms as it works towards making the system highly reliable.