I really want to love this, but my experience in the first 20 seconds is unfortunately like some of my other experiences coding against Fly APIs, they're broken.
can I live with some rough edges for some personal workflows that only impact me when things break? sure. however, I was thinking about playing with some CI/CD stuff using sprites that would impact our whole team if things broke and I'm really on the fence because of this experience in the first 20 seconds.
Fly team - please put some black box probes or just better testing on the example you give in the quick start. if you document it, test it.
Recording voice overs mostly. My directional mic is picking up a good bit but maybe I'll try another one just to be sure. I've got this cheapo right now that worked pretty well on my old space - https://a.co/d/gUmdYpM
That and some drapes will be what I try this weekend, maybe next weekend try to add some acoustic panels to the wall right in front of my desk or something.
I appreciate it's a good bit of guess and check. I was hoping to avoid that and get to the root of it with a technical acoustics answer but I also appreciate there's people with phds in this kind of thing so something something balance.
there is a good chance that redundant streams would have helped here assuming that the player could load the manifest to fail over, but that manifest sadly hits Fastly. the problem with redundancy is that it really only works if you are redundant all the way up and down the serving stack. in our standard serving path (even without redundant streams) we automatically select the optimal CDN for chunk serving which does take into account availability, but during this incident we measured Fastly as about 5% less available than normal which wasn't enough to trigger a full automatic drain off of them. turns out, a 5% failure rate when you are serving 50 chunks to enable a video playback means 1 of them will likely fail and you get a playback error or in the best case, rebuffering.
sadly, Mux's manifest servers have not been redundant because the edge logic is a pile of Fastly's VCL. as mmcclure says, we're working on adding another CDN that can run our manifest serving logic so that we can be fully redundant.
I am really looking forward to a standard and portable edge platform that makes redundancy at this application layer easier. wasm here we come?
I've been working on a project with a couple people that I don't get a lot of value from as co-founders right now because they aren't designers or engineers. They are trying to help us define the set of requirements given their relationships in the industry, but they have not really done product work before. I keep telling myself how valuable they will be once we launch because they will bring customers. Echoed by the author here:
"When you start a project, you always think that finishing and launching your project is the hardest part. However, after the launch you realize that the most difficult part is just ahead."
Any recommendations for how to handle this kind of co-founder situation?
I'm thinking about working with them to define equity incentives on both sides, but I'm short on other ideas for how to make things feel well balanced in the short term given our unbalanced skill sets.
Matt Ward - engineer at Mux here. The simplest answer here is that most browsers and native platforms don't easily support RTP/RTSP these days. It isn't technically impossible to stream RTSP to say an iPhone, but it I'd say it is non-trivial compared to simply using AVPlayer with HLS. Our goal with this product is to make video easier for developers to understand and use so that they can focus on the other parts of their apps. We are running those rather complex streaming servers and we even provide some guides for how to get started streaming video to common platforms - https://docs.mux.com/docs/playback
If you have questions about video streaming on a specific platform feel free to reach out! (my full name at mux)
Matt Ward - engineer at Mux here. I'll address a few of your concerns. First off, I worked on building YouTube's video infrastructure for years and now I'm working on Mux's with people that built Zencoder. So our team has seen a few things, and rest assured my friend we know video is hard. That is actually part of why we started simple with just HLS which does get decent browser coverage and its works well on Android with ExoPlayer and iOS as well which a lot of our customers care about. What would you have started with?
Curious where you saw we were using the nginx rtmp module given that we don't have a public live streaming offering yet, but it is coming soon! Oh and yes nginx rtmp is probably the best open source offering on the market now so we will be using that. We've got plenty more coming as well so stay tuned.
If you want to select the quality you want to view at, that's a player decision. You can bring your own player to Mux. We think that's pretty neat. You can't bring your own CDN though, because we've taken that part of out of the equation for you. We've actually started integrating with multiple CDNs and have a Cedexis integration as well which helps us pick the optimal CDN for each playback. Hopefully handling all that configuration helps make things easier for developers just getting started with video and enables better smoother playbacks for everyone.
Finally, we are hiring so if you know some things about video and want to join us please reach out!
it has a more visual explanation of the OSI model and how it relates to routing and different kinds of hardware. I also tried to explain some of the interesting problems in actually building out a network in the second half of my talk.
if anyone is just trying to learn the basics of networking I'd also strongly recommend the Juniper Networking Fundamentals online class, its free at https://learningportal.juniper.net/juniper/user_activity_inf... or you can find videos of it on YouTube.
It is important to note that message complexity has a lot to do with proto performance. Anyone with a basic understanding of algorithms could tell you that, but I prefer to be shown things so here's a repo with some examples of Python proto serialization performance so you can see how it does as message complexity increases:
So yes, if your application has to send a lot of structured data around, you will end up paying a non-trivial serialization cost. On the other hand, if your application is sending really simple messages then I highly doubt proto serialization is the performance bottleneck you should focus on.
Here are a few things I highlight in this post as things to consider when architecting for failure: Retry, Backoff and Rate Limit. Use a Cache. Add Redundancy. Build a Buffer. Reconsider Dependencies. Introduce Isolation. Improve Test and Release Practices.
Click the post for more about how I think about each of these. I think that considering cost tradeoffs when doing evaluating each of these approaches is what makes architecting systems so challenging (and interesting).
What else do you do in your systems to handle failures gracefully? Any questions about what we are doing or how we are doing it?
a "quick start" really should just work when you copy paste them.