When Amazon CEO Jeff Bezos unveiled his company’s long-term plans to fulfill customer orders by way of same-day drone delivery back in 2013, it seemed to many like a science-fiction idea come to life. More than a decade later, the retail giant continues to hover closer toward its futuristic objectives, with limited drone deliveries already taking place in two select metropolitan areas in Texas and Arizona.
As a software development engineer at Amazon Prime Air, the research and development division responsible for drone delivery, 2007 DigiPen graduate Jonathan Wills is part of the team working to make the company’s automated technology safer, faster, and more reliable — all with the goal of bringing its new mode of service to more customers and locations down the line.
“My current work is in the development of our web-based tooling for both 2D and 3D data annotations based on the collected flight data,” Wills says. “The output from these manual annotations is then fed back into our org’s machine learning to improve the autonomous flight of our drones.”
As with any application of autonomous vehicle technology, Amazon’s drone delivery relies on a combination of computer vision and machine learning systems. On the one hand, you have the drone itself, which contains a series of built-in sensors and cameras that allow the drone to “see” its surroundings as it flies and maneuvers. In order to make sense of those surroundings, however, the drone’s flight systems must be trained to correctly identify what it’s seeing — from power lines and water towers to living beings like birds and pets — and react accordingly.
That’s where human intervention plays an important role, according to Wills. Taking the drone’s camera captures post-flight, teams of data annotators are able to highlight objects within the frame of those images, using customized software built by Wills’ team, to improve the drone’s artificial intelligence navigation systems.
“Because it needs to learn, there’s a manual effort of labeling to go through and look at the different outputs,” Wills says, referring to the 2D camera captures. “What people will do is they’ll identify things — even if they’re really tiny or in the background. They’ll go in and label an airplane or any other sort of thing, like a balloon.”
In addition to the 2D image labeling, Wills says, Amazon’s data trainers also make annotations in 3D space, using a web-based tool to point out objects from another perspective.
“For the 3D side, we have a pipeline that will take those same incoming images, and then we convert them algorithmically into a 3D point cloud,” Wills says.
All of the annotated data, in turn, goes back into Amazon’s machine learning algorithms, resulting in better and more optimized autonomous decision making over time.
Graduating from DigiPen is a tremendous source of pride for me. It is a great school that ensures you learn how a computer system actually works in very low-level ways.
For Wills, who joined the Prime Air team last January, the job marks a culmination of the many engineering skills he’s been mastering and accruing since his days in the BS in Computer Science in Real-Time Interactive Simulation program at DigiPen, which he credits for laying the foundation that has supported his 18-year career.
“Graduating from DigiPen is a tremendous source of pride for me. It is a great school that ensures you learn how a computer system actually works in very low-level ways. Starting off with C and C++ also makes any other C-style language easier to learn,” Wills says. “I’ve found that I can pick up complicated concepts more easily.”
Today, Wills works closely with an interesting mix of fellow engineers, including ex-game developers, traditional full-stack developers, and several people from the aerospace industry.
“This role is really neat because it is a mixture of nearly everything I’ve done since school,” he says. “I get to stretch my real-time interactive software muscles, as well as very full-stack work such as web front end, back-end cloud service implementation, and infrastructure architecture.”
A turnaround rendering of Amazon’s MK30 delivery drone.
After starting his post-DigiPen career in game development, it wasn’t long before Wills moved on to other areas. While his prior work at places like Microsoft and Amazon Web Services (AWS) saw him branching out into other areas of tools development and web-based software engineering, Wills says he always found himself enjoying the chance to pivot back to 2D and 3D application development — even for non-game uses. That interest was especially rekindled during one of his assignments at AWS, where he worked prior to joining the Prime Air team, when he helped build a 3D editor for making digital twins of internet-connected objects and devices.
“It was the first time since my early career that I was back to working on real-time 3D software, and it made me realize just how much I missed the work,” Wills says. “So I always kept an eye out for interesting positions within Amazon, especially those that involved anything to do with 3D software. One day I happened across this open position that needed someone with 3D expertise and jumped on it.”
As luck would have it, the manager for the position was fellow DigiPen graduate Scott Smith (2005, BS in Computer Science in Real-Time Interactive Simulation), who liked the idea of bringing someone with Wills’ educational and professional experience to the team.
“When I reached out to him about the position, he messaged me back in a couple minutes,” Wills says. “We immediately hit it off and knew that it was going to be a great synergy and that I’d be a good fit for the role.”
So far, Wills says, his varied skillset has indeed come in handy for developing the various tools used for data training. In addition to being a resident technical expert on the 2D and 3D mathematics inherent to the graphical rendering of his team’s software applications, he also regularly tackles the problem of how to make said applications more optimized and performant — even for less powerful computers.
“Due to the large nature of our datasets, a traditional 3D web application would not be suitable for a wide variety of machines. So I work with both the back-end server-side rendering of 3D and the web interface, along with the streaming of input and output between the two interfaces,” Wills says.
How quickly the company will be able to achieve its ultimate goal of widespread 60-minute drone deliveries remains to be seen. At the moment, the company is purposely focusing on dry climate areas with more-or-less predictable weather patterns — as well as limiting drone-carried packages to five pounds or less.
“You want to avoid places that are going to have volatile weather, such as too much rain that can blur the cameras or things like too strong of winds,” Wills says. “The idea is we get the controlled environments working well first and then expand.”
Just like the Amazon drones that continue to learn and improve thanks to the power of machine learning, Wills similarly encourages aspiring game and software developers to embrace the process of continual growth and development.
“I’ve really enjoyed getting experience in various areas of software development. It is one of those ways to keep yourself from getting bored but to also find what you enjoy most,” he says.