Researchers who created the technology behind Microsoft’s Photosynth have upgraded their software to reconstruct entire cities in digital 3D by processing photos downloaded from the photo-sharing Web site Flickr.
A team at the University of Washington used 150,000 photos tagged with “Rome” or “Roma” from Flickr to create a 3D model of Rome in 21 hours. It has also reconstructed Venice, Italy from 250,000 images in 65 hours and Dubrovnik, Croatia from 60,000 images in 23 hours. The processing was done using a cluster of 350 computers.
Improving on an earlier photo-stitching technology that was designed to create a 3D model of a room or a monument from photographs, the team made the code 100 times faster by refining the way overlapping photos are matched to one another and adding the ability to do processing in parallel on many computers, or remotely via the Internet.
“That work was essentially aimed at landmarks, involving a couple hundred to a thousand images,” says Sameer Agarwal, acting assistant professor of computer science and engineering at the University of Washington. “When you have hundreds of thousands of images available, there’s no way you can think of doing all of those comparatively, even if you had the hardware.”
Microsoft Corp. provided some funding for the University of Washington project. The software giant licenced Photo Tourism in 2006 and used it as a basis for Photosynth, a tool that is now online and freely available. Users can navigate the environments created by others – or download software to create their own 3D environment from a collection of photos.
Photosynth is just over one year old and took 20 months to take from research to a product, says David Gedye, Photosynth Group Program Manager at Microsoft.
The team is interested in expanding its project to allow for a larger experience.
“These guys, at least from a research perspective, have sold the problem of a very large scale synthesis,” Gedye says. “They’ve solved the problems nobody else could solve.”
To trim down on processing time, researchers designed an algorithm that would do a quick-and-dirty comparison between images, and identify likely candidates that included an overlapping area. The software then hones in on specific sections of an image where an overlap area is found, and does a deeper comparison.
Writing the algorithm so it could be run across a number of computers also improved the speed of the process.
Video and images of the 3D reconstructions show cities that appear to be assembled out of billions of dots. The images resemble an artistic technique known as pointillism.
“These are collections of points right now,” Agarwal explains. “The next stage of processing is creating dense mesh models.”
Such a technology could have many different applications, he adds.
“You could use it for next-generation GPS, or use it for augmented reality,” he says. “There’s also an academic aspect to it, imagine if you have a full 3D model of a city, you can compare it to other cities and see what is similar.”
The technology could also be used for entertainment production, such as a virtual reality movie set or an environment for a video game. Imagine playing Grand Theft Auto: Rome, for example.
Photosynth has found some big name users, Gedye says. The Smithsonian has used it to show off exhibits online, and the technology has even been used in Earth’s orbit.
“NASA has trained astronauts who go to the International Space Station to take Photosynths,” he says. “They’ve Photosynthed many modules of the space station, and they’ve done it for the new Mars rover.”
The hospitality and real estate industries are also prime candidates for use of Photosynth in online marketing, he adds.
Photo Tourism provided more the concept of creating 3D digital models from photographs than it did the code for Microsoft, Agarwal says. The Photosynth team has rewritten much of the code base since licencing the technology.
“Research code is not necessarily production-worthy code,” he says.
Microsoft isn’t actively pursuing this piece of research for commercialization yet, Gedye says. But it’s an interesting application to consider.
“Scaling up in size and allowing small Photosynths to link together and create larger experiences is something we’ve been wanting to do for awhile,” he says. “We’re certainly interested in it.”
A research paper based on the University of Washington technology will be presented in October a the International conference on Computer Vision in Kyoto, Japan.