I write fairly deep ML performance tuning articles at https://paulbridger.com and the (many) hours I've spent on each article have been hugely worth it.
Many people reach out to me via this work, and when we talk they already see me as an expert or already want to work with me.
BTW, this is pumping the same video file through the network - not just a single file.
I don't measure latency, but this is not a deep pipeline so it's easy to calculate.
Great question, now I wish I'd recorded power consumption for all these experiments.
Judging from cumulative hours of watching the output of nvidia-smi I've definitely seen a linearish relationship between utilization and power draw (with a non-zero floor of 30-40W).
Very practical question :)
Exactly as you say, multi-stream throughput. Also for faster than realtime offline processing of video.
Check the caveats section at the end of the post - DeepStream is probably not well suited to high throughput single-stream inference.
To me, seeing the GIL held for 40% of time and significant time spent waiting on GIL by other threads was a fairly strong indicator. Keen to hear your thoughts/experience on it.
Great point - dependencies between frames are inherently problematic for many of these techniques.
Everything lostdog says. I've had experience speeding up tracking immensely using the same big hammer I talk about in the article - moving the larger parts of tracking compute to GPU.
Also, in a tracking pipeline you'll generally have the big compute on pixels done up front. Object detection and ReID take the bulk of the compute and can be easily batched and run in parallel. The results (metadata) can then be fed into a more serial process (but still doing the N<->N ReID comparisons on GPU).
Author here. As other commenters are saying, the Pytorch JIT and torchscript might be your friend here.
Alternatively, there are some quite fast OSS libraries for object detection. Nvidia's retinanet will export to a TensorRT engine which can be used with DeepStream.
It’s important to note this is just within visual range dogfighting/BFM. Also just guns, and also perfect enemy state information.
Important progress but it’s super early. Beyond visual range/BVR, coordination with wingmen and many other higher-level tasks still to come (or are still secret).
Similar thing happened to me. Lost access to a perfectly set-up forwarding account. The account recovery process is impossible because I nolonger have the same phone number from 10 years ago.
I think I understand your point about declaring and then later using your layers.
Are you aware of the Sequential module? It allows you to chain together layers into a single variable, making this repetition disappear into a single forward/__call__ on the Sequential.