I’ve been working on open sourcing a background jobs library I built for Rust and Postgres, called Underway.[0] Unlike other similar queuing libraries, it offers a simple “step” functions API for defining dependent units of work.
I built this because a number of projects I work on need a robust, resilient way of deferring work but I didn’t want to add another piece of infrastructure or another language to my stack. Plus as soon as you start to reach for APIs that offer some kind of workflow concept, your options become fewer and further between.
> but the 800-pound gorilla in the room is climate change. Where I live, winters are 4.5 degrees warmer.
The George W. Bush administration (e.g. via Frank Luntz) advocated for the term "climate change" because Republican strategists wanted to leverage perceived uncertainty about global warming as much as possible.[0]
This is a PR effort that seems to have largely succeeded (both in adoption and its goals) and it's unfortunate that when we are literally talking about warming we adopt a term that is less precise; you are talking about global warming here.
While I understand your point is likely that sidewalks would come before later advancements (like support for bikes) I want to make it clear that bikes do not belong on sidewalks.
If you ride a bike please ride it in the street with other vehicles. This is the law in some jurisdictions (such as where I live) but frequently ignored.
Failure to do this poses a serious risk to pedestrians. Please do not use sidewalks as an alternative road.
I'm coming around to building everything for a VPS from the outset. There's a lot of upside to VPSes, such as:
1. Can be purchased as a fixed cost, usually at a rate that's much cheaper than on-demand pricing, and especially serverless--this tends to only get better with time as competition keeps prices low
2. It's "just" a Unix/Windows/Mac box, so the issues with runtime constraints you mention are bounded differently (and often more favorably); serverless is also just a box, but the constraints tend to be more onerous and limiting and it's not usually accessible in the same way
3. With containers, it's trivial to move between providers, so the hardware itself becomes fungible
4. On containers, I'm having a great time shipping Docker Compose configs--this works really well for the scale of application I'm targeting while avoiding the dreaded complexity of e.g. k8s
5. There's decades of high quality tooling already built and battle tested which makes operating VPSes much easier; the fact you can SSH into the machine, for instance, has huge leverage as an solo person working on independent products
Going forward, I'm planning to skip edge compute altogether unless there's a really compelling reason to want it. I should also mention that when a VPS is paired with a CDN, you can layer on bits of "edge compute" where it's warranted; or, you know, use it to cache static assets close to your users. :)
All-in-all it's kind of a funny return to where I started ~20 years or so ago with web development.
This is incredible. If I understand the thread, that >$22,000 surprise bill was not forgiven by Vercel and the most they're willing to do is offer a 25% discount.
Is Vercel a business or a scam masquerading as a tech company?
If a company needs to stoop to this level of billing shenanigans to make money, I have my doubts...
In fact Lee was looped in six weeks ago; there was an issue with Remix and CDN cache behavior which popped up on GitHub. I mentioned my issues in that thread thinking they were likely related. Lee responded and let me know he would talk to the internal team.[0] However, Lee did nothing that was ever visible to me. Moreover, I completely disagree with your assessment: it should not require Lee or social media posts to tackle issues like this, that’s completely unscalable and not a realistic way to run a business.
It’s pretty irresponsible of you to be suggesting folks reach out directly to Lee when frontline support fails. That sucks for Lee and it’s bad for the Vercel business.
To be blunt with you, your comment does not read well. It looks like you’re ignoring very real problems and doing everything you can to dismiss them as “bad faith” when in fact there’s a demonstrable problem which shouldn’t be excused as growing pains but instead addressed head on. This kind of fanboyism doesn’t help anyone and I hope Vercel takes the time to reflect on this feedback and make real, meaningful changes.
Actually I reached out to you asking for a comment which I could add to the article, not for your technical support.
I was surprised and I have to admit a bit dismayed to watch you throw yourself into the fray on a Sunday. The technical issues, which in fact persist, are at this point an aside to the way Vercel has handled this issue.
I'm currently stuck in their support hell and have been told:
1. My issue is not real
2. Okay, your issue is real, but because I'm not paying $$$ we're going to ignore you
3. I should do free work for Vercel and poll their community forums to see how widespread the issue is
4. Their support is only trained to handle frontend issues and because this is an issue with their CDN it's expected that they'll respond incompetently
5. They'll escalate with their CDN team and respond in one week (that was over a month ago, no follow up whatsoever)
It's hard to take Vercel seriously. As a toy, it's probably fine. But I'll ultimately move this project off of their CDN product as soon as it reaches costly volume.
The fact is unless these models are not what they promise to be, there's no way of weeding out generated content from not without meaningful false positives.
On the other hand, if the models are in some way deterministic, then it stands to reason there's some plausible method of detection.
But if that's the case then there's a tremendous flaw in the foundation of these techniques...they aren't at all what's been promised.
So far that doesn't seem to be the case.
Finally it's worth pointing out that Google et al really shouldn't care if content was generated by a machine or not. The problem isn't content creation, it's content quality. So what Google actually cares about is delivering a result that meets a certain quality threshold. It's very difficult to believe Google cares much about the process of content creation.
To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products.
One of the most significant pitfalls of data is failing to interrogate the value it provides and assuming that if you give everyone access all the time the magic will happen. The truth is value does not simply materialize just as value does not magically spring from computers by a human powering it on (okay sure, you may have already automated the value but that’s actually the point I’m about to make). In both cases it requires an experienced practitioner who collaborates with a larger team to intersect their work with the business needs.
Data is tricky, all the more so because it’s often seen as a panacea by business leaders who aren’t connected with the work of extracting that value.
It's a great question: fundamentally the Parquet format offers columnar orientation. With datasets like these, there's some research[0] indicating this is a preferable way of storing and querying WARC.
DuckDB, like SQLite, is serverless. Duck has a leg up on SQLite though when it comes to Parquet: Parquet is supported directly in Duck and this makes dealing with these datasets a breeze.
Curology | Engineering Manager, Platform Engineer, Data Engineer, and many others! | San Francisco, CA | Full-Time, Onsite
Engineering Manager
Curology's Platform Engineering Team is looking for a passionate, experienced, and creative data engineering manager to lead, grow, and mentor the Data Platform Team. As an Engineering Manager of the Data Platform Team, you will work closely across teams to design and implement the pipelines and infrastructure that power our data science, business insights, and marketing efforts. The perfect candidate will have strong data infrastructure and data architecture skills, a proven track record of leading and scaling engineering teams, strong operational skills to drive efficiency and speed, strong project management leadership, and a strong vision for how data can proactively improve companies. This is a full-time position based in our San Francisco office.
Curology | San Francisco, CA | DevOps and Data Engineers | $130k-$175k | FULL-TIME | ONSITE | https://curology.com
ABOUT US: Curology is a telemedicine startup focused on making dermatology accessible to everyone. We're growing quickly—300% in the past year alone—and creating excitement by helping real people see life-changing improvements in their skin.
HIRING: We're looking for talented devops and data engineers to lead development of our infrastructure and data engineering efforts. There are a lot of interesting and challenging problems to solve for both roles--scaling and privacy are top-of-mind.
BENEFITS: Relocation, competitive salary, meaningful equity, free food, health insurance, open vacation policy
These are two separate roles and the details of each can be found on AngelList:
I built this because a number of projects I work on need a robust, resilient way of deferring work but I didn’t want to add another piece of infrastructure or another language to my stack. Plus as soon as you start to reach for APIs that offer some kind of workflow concept, your options become fewer and further between.
[0]: https://github.com/maxcountryman/underway