Perceptiv’s Shift (YC W15) Turns Drones into Smart Dollies for Filmmakers(blog.ycombinator.com)
blog.ycombinator.com
Perceptiv’s Shift (YC W15) Turns Drones into Smart Dollies for Filmmakers
http://blog.ycombinator.com/perceptivs-shift-yc-w15-turns-drones-into-smart-dollies-for-filmmakers
17 comments
Founder here :).
Thanks for the great comments!
OpenCV has very rudimentary tracking capabilities. This is why we went back to the drawing board and developed something that is robust in a wide number of cases including in busy environments or where saliency is not high.
You are correct about tracking reliability, however, most of our early customers were more worried about creative control and smoothness. So, we've worked hard to make sure they get that level of control while not having to worry about the minutiae of controlling a drone/gimbal.
In cases of occlusions or odd situations where it may lose track of the actor, they are more than happy to quickly re-initiate tracking.
You are correct about tracking reliability, however, most of our early customers were more worried about creative control and smoothness. So, we've worked hard to make sure they get that level of control while not having to worry about the minutiae of controlling a drone/gimbal.
In cases of occlusions or odd situations where it may lose track of the actor, they are more than happy to quickly re-initiate tracking.
Heck yes! Alas not a filmmaker but I have wanted something very nearly like this forever. I go horseback riding regularly, mostly through relatively open areas (county parks / regional preserves / city trails etc). I have wanted a drone that can carry a GoPro (or something) to capture footage of our rides (sometimes we go through some gorgeous country that we really wish we could capture on video).
However, here is the kicker, because we ride at a fast trot or canter, I can't actively control the quad. Instead, I must be able to passively control it. It must perform specific, pre-defined actions based on my position, speed, and heading while flying, steering, and maneuvering without receiving any input directly from me. It should continually adjust its position and flight path based on my speed according to the rules I pre-define. For example, after takeoff with the camera turned on, the drone ought to stay ahead of me, say 30 feet ahead and 20 feet off the ground. Then if I start moving faster than 10 mph it should come to my side about 30° (10 o'clock). If I continue at this speed for more than a minute it should do a slow semi circle around me and come to the 2 o'clock point for a minute. If I keep going it should... etc etc. I don't care if I have to program the drone in code or through a UI, I just want to be able to define lots and lots of rules (based on the terrain of the day). Of course it would need to protect itself from running into trees and walls, or any other objects while trying to follow my rules as closely as possible.
You have my attention.
However, here is the kicker, because we ride at a fast trot or canter, I can't actively control the quad. Instead, I must be able to passively control it. It must perform specific, pre-defined actions based on my position, speed, and heading while flying, steering, and maneuvering without receiving any input directly from me. It should continually adjust its position and flight path based on my speed according to the rules I pre-define. For example, after takeoff with the camera turned on, the drone ought to stay ahead of me, say 30 feet ahead and 20 feet off the ground. Then if I start moving faster than 10 mph it should come to my side about 30° (10 o'clock). If I continue at this speed for more than a minute it should do a slow semi circle around me and come to the 2 o'clock point for a minute. If I keep going it should... etc etc. I don't care if I have to program the drone in code or through a UI, I just want to be able to define lots and lots of rules (based on the terrain of the day). Of course it would need to protect itself from running into trees and walls, or any other objects while trying to follow my rules as closely as possible.
You have my attention.
Wow. This is great information! You have a very well laid our requirement :).
"Of course it would need to protect itself from running into trees and walls, or any other objects while trying to follow my rules as closely as possible."
And that is the kicker. A lot of the features you've outlined can be done with current technologies simply using GPS (though framing would be wrong unless you have a SHIFT-like product). However, none of it is safe to use. Even if you were in "mostly" open areas, to constantly watch out and make sure the drone doesn't crash is very cumbersome.
Comprehensive sense-and-avoid is the golden egg of the drone community, precisely because of the features you've mentioned!
"Of course it would need to protect itself from running into trees and walls, or any other objects while trying to follow my rules as closely as possible."
And that is the kicker. A lot of the features you've outlined can be done with current technologies simply using GPS (though framing would be wrong unless you have a SHIFT-like product). However, none of it is safe to use. Even if you were in "mostly" open areas, to constantly watch out and make sure the drone doesn't crash is very cumbersome.
Comprehensive sense-and-avoid is the golden egg of the drone community, precisely because of the features you've mentioned!
Nice application! Based on what I have seen so far, I am pretty sure they are using a tracker based on correlation filters [1]. These are very fast and robust. Naive implementation does not provide scale, which is visible in the video. but this is rather trivial to do. OpenCV 3.0 has also implemented this MOSSE [2] tracker.
Overall, these trackers are not able to learn and will always fail in same situation. It would be great if they had re-detection capability and improve with every touch from the operator.
[1] http://home.isr.uc.pt/~henriques/circulant/ [2] http://www.cs.colostate.edu/~bolme/publications/Bolme2010Tra...
Overall, these trackers are not able to learn and will always fail in same situation. It would be great if they had re-detection capability and improve with every touch from the operator.
[1] http://home.isr.uc.pt/~henriques/circulant/ [2] http://www.cs.colostate.edu/~bolme/publications/Bolme2010Tra...
Wow! The author of TLD, that's quite the honor :).
You are right about a lot of the issues we've been facing. However, our focus is squarely on getting tracking results VERY fast, so we can get smooth operation from our controllers. We are working hard to incorporate some global re-detection (and scale), however, we are balancing this with running the algorithm fast online, onboard the drone.
I'd love to talk and bounce more ideas off of you! Ping me, pj(at)perceptivlabs.com
I'd love to talk and bounce more ideas off of you! Ping me, pj(at)perceptivlabs.com
I wonder if this is based on OpenCV or if it's a proprietary system. Looking at the demo it could be using CAMShift with Kalman control?
My experience with OpenCV is that object tracking depends heavily on hue differences between the tracked object and the background - eg. I start out tracking my face, and wave my hand in front of my face, the tracker then starts to track my hand instead. Have you found a good way to mitigate this issue?
My experience with OpenCV is that object tracking depends heavily on hue differences between the tracked object and the background - eg. I start out tracking my face, and wave my hand in front of my face, the tracker then starts to track my hand instead. Have you found a good way to mitigate this issue?
Very good question!
a) We've found that OpenCV's boilerplate algorithms have much bigger problems than the ones you have mentioned in deployment scenarios. So, we've had to develop something better from the ground up (while still using the OpenCV infrastructure).
b) For the tracker to go back to tracking your face, it would require global re-detection which is usually a much slower process. Moreover, the tracker would have to know what your face looks like under different lighting conditions and angles. This is something we are investing in heavily. Convolutional neural networks have shown a lot of promise!
This is fantastic. I've been wanting to see something like this!
I'd love to see a partnership with GoPro and get these into live events (once GoPro can do live feeds). I'd love me some Red Bull Rampage, or Supercross footage.
Supercross has this spider cam concept they started this year, but it still requires a camera on wires zigzagging the stadium, this would solve the issue. if you can get live footage out.
awesome stuff!
I'd love to see a partnership with GoPro and get these into live events (once GoPro can do live feeds). I'd love me some Red Bull Rampage, or Supercross footage.
Supercross has this spider cam concept they started this year, but it still requires a camera on wires zigzagging the stadium, this would solve the issue. if you can get live footage out.
awesome stuff!
Perceptiv founder here. Thanks for the support! Would love to talk to GoPro but we haven't been able to do that, so please talk to us if anyone has leads :) Great idea about live streaming! We're definitely trying to figure out how to do that.
Filmmakers?
This is law enforcement and military's wet dream. Chasing suspects or hostile force in an urban setting. Possibilities are endless.
This is law enforcement and military's wet dream. Chasing suspects or hostile force in an urban setting. Possibilities are endless.
Hahah! I'm sure it is, but it is far less interesting to us. To know that someone is creating fantastic works of art using something you've made is much more rewarding :).
Never thought we could possibly have self flying video camera drones that I used to see in animation films, in real life...
amazing product and kickass team
I wonder how they do tracking. The tracker in OpenCV isn't that good, but then it's about fifteen years old.
Movie production will soon be using fewer Louma cranes and manned helicopters.
They'll eventually need to put an ADS-B transponder (there's a 100g unit available for drones) on the thing and get the procedures in place with the FAA so productions can use this in areas near airports. That makes the drone radar-visible to air traffic control and other aircraft.