emef · HackerTrans

emef

no profile record

Submissions

Former Expedia CEO Dara Khosrowshahi Is Officially Uber's New CEO

theverge.com

1 pointsby emef9년 전0 comments

A note on our lawsuit against Otto and Uber

medium.com

1 pointsby emef9년 전0 comments

Line-based fractal images in go

github.com

3 pointsby emef12년 전0 comments

comments

3년 전discuss

The model weights (the thing being updated by the training process) stay loaded in gpu memory during training (the slow part). This could be useful to serialize the model weights to disk when checkpointing or completed, but it's a drop in the bucket compared to the rest of the time spent training.

4년 전discuss

it was an IoT product (internet-connected LED sign)

4년 전discuss

how are compilers not performance-critical applications?

4년 전discuss

I did a proof of concept in luigi pretty early on and really liked it. Our main concerns were that we would have needed to bolt on a lot of extra functionality to make it easy to re-run workflows or specific steps in the workflows when necessary (manual intervention is unavoidable IME). The fact that airflow also had a functional UI out of the box made it hard to justify luigi when we were just getting off the ground.

4년 전discuss

It's a good question. I believe airflow was probably the right choice at the time we started. We were a small team, and deploying airflow was a major shortcut that more or less handled orchestration so we could focus on other problems. With the aid of hindsight, we would have been better off spinning off our own scheduler some time in the first year of the project. Like I mentioned in my OP, we have a set of well-defined workflows that are just templatized for different jobs. A custom-built orchestration system that could perform those steps in sequence and trigger downstream workflows would not be that complicated. But this is how software engineering goes, sometimes you take on tech debt and it can be hard to know when it's time to pay it off. We did eventually get to a stable steady state, but with lots of hair pulling along the way.

4년 전discuss

We've also been running airflow for the past 2-3 years at a similar scale (~5000 dags, 100k+ task executions daily) for our data platform. We weren't aware of a great alternative when we started. Our DAGs are all config-driven which populate a few different templates (e.g. ingestion = ingest > validate > publish > scrub PII > publish) so we really don't need all the flexibility that airflow provides. We have had SO many headaches operating airflow over the years, and each time we invest in fixing the issue I feel more and more entrenched. We've hit scaling issues at the k8s level, scheduling overhead in airflow, random race conditions deep in the airflow code, etc. Considering we have a pretty simplified DAG structure, I wish we had gone with a simpler, more robust/scalable solution (even if just rolling our own scheduler) for our specific needs.

Upgrades have been an absolute nightmare and so disruptive. The scalability improvements in airflow 2 were a boon for our runtimes since before we would often have 5-15 minutes of overhead between task scheduling, but man it was a bear of an upgrade. We've since tried multiple times to upgrade past the 2.0 release and hit issues every time, so we are just done with it. We'll stay at 2.0 until we eventually move off airflow altogether.

I stood up a prefect deployment for a hackathon and I found that it solved a ton of the issues with airflow (sane deployment options, not the insane file-based polling that airflow does). We looked into it ~1 year ago or so, I haven't heard a lot about it lately, I wonder if anyone has had success with it at scale.

4년 전discuss

parquet is great but it's not particularly easy to read or write. the libraries that do exist to work with it are few and far between, and those that do either have a hundred dependencies or depend on native code (e.g. libarrow). certainly an important dimension in an ideal file format should be the ease of parsing/writing it, and parquet gets an extremely low score on that front imo

4년 전discuss

core to tesla's strategy is to do massive data collection from consumer-owned cars using beta software (and hardware, that the consumer pays for). that model is not compatible with expensive lidars, which contrary to some other comments in this thread, are still very expensive (just because the entry-level pucks are cheap, does not mean full lidar coverage is cheap). there is no way they could push $100k of sensors on consumers to build out their data collection pipeline. when tesla was first starting out, affordable lidar did not even exist so it's hard to call that a lame excuse.

all that said, I'm still pessimistic about tesla's chances at making camera-only L4 work in any short time horizon. we will see if they pull it off, but it's such a severe disadvantage compared to fully-kitted competitors.

5년 전discuss

> Here's a fair warning: this article is reductio ad absurdum,

literally the first sentence...

5년 전discuss

SEVerity level of the incident

5년 전discuss

this is less true these days w/ slim docker images that don't include any editors by default

5년 전discuss

  # does nothing when an update is tried 
  frozen_shared_state = freeze(shared_state,
  frozen_shared_state.count = 4  # Does nothing, as this update did not exist

yikes. Thoughts on when this feature would ever be useful? Just the thought of working in a codebase with this subtle inconsistency makes me cringe.

6년 전discuss

I wrote our internal lightweight version of neuropod at another SDC startup where we did use TensorRT. Our ML researchers worked in pytorch and more often than not, the pytorch -> onnx -> tensorrt conversion did not work. We ended up needing to replicate the network architecture using the tensorrt library and manually convert the weights from pytorch. Then we'd use the tensorrt serialization to compile the models so they could be run in c++. I imagine that they may have tried this in neuropod and saw the same conversion problems. TensorRT was a big investment to get running smoothly but it did shave off 20% or so off our inference latency

6년 전discuss

I track account balances once a month into a spreadsheet to keep a high-level overview. It's a manual process that takes ~15 minutes and has a large payoff to have a birds-eye view of all the finances and allocations.

Investments are all automatic: max 401k, max hsa, auto deposit to vanguard into a few indexes. If cash balance gets too high, I'll do a manual deposit.

My wife and I aren't big spenders so I have never found much value in micro-managing individual purchases or categorization (YNAB or mint). I do skim the transactions a couple times a month to make sure there aren't any surprises.

wrt paying off loans vs investing: lots of approaches here, the emotionless take is to just allocate based on interest rate. If your loans are low interest, then just pay the minimums and invest the rest. If they're higher interest than you expect your returns from investing to be, then just pay them off aggressively and don't invest at all until they're paid off (the exception here might be tax-advantaged accounts like 401k or IRA if you qualify).

7년 전discuss

Had to wait forever to get a time slot. They do 10-15 minute demos in a temporary room they put up on the expo floor.

7년 전discuss

I tried their demo at CVPR a few months ago. It was super immersive and crisp. You interact with an AI "guide" who shows you how to look around, pick things up, stick them on the wall, etc. It was surprisingly unnerving to look the AI in the eyes and have her react to my gestures. Beyond a sweet demo, not sure what the value in the product would be for a general consumer option, but cool technology for sure.

7년 전discuss

Transaction cost and speed are enormous problems for bitcoin, not sure why that would only be mildly appealing to solve. Not vouching for Libra, but my understanding of bitcoin was that it caps out at something ridiculously low like 11 transactions/second, which each cost multiple USD. Clearly that is the bottleneck that needs to be addressed by whatever would lead crypto moving forward (I remember reading visa processes more than 10k transactions/s).

7년 전discuss

I'm not sure what kind of deployment you are envisioning where this makes sense. If you are running a java application, you likely want a bare bones OS + jvm of appropriate version (like the ones being discussed).

7년 전discuss

This is why we use a redundant geometric approach (with LiDAR) as a fallback to NNs in our perception stack. NNs can have false negatives which is unacceptable, but dealing with false positives is more manageable and safer.

7년 전discuss

Tesla has take a firm stance against using LiDAR and is doubling down on CV. I'm not sure if they have radars for parking assistance, but they are using cameras as primary sensor for perception and localization.