Looks pretty great! The free tier also looks reasonable. The pricing on the other tiers isn't outrageous either if you use it consistently. Unfortunately, I likely find myself in the big gap between the free tier and the Basic plan. I can't justify yet another subscription that I use only a couple of times a year. That said, I would happily pay the $6 on the months that I use the service. Given the churn issues, I'm surprised more SaaS offerings don't work that way.
Not sure if you found this as well, but gpt-3.5-turbo-016 does a poor job following instructions other than parsing. So, to work around this, we prompt gpt-3.5-turbo with the rules we want applied say an extracted field and then go back to gpt-3.5-turbo-016 to parse with chatgpt functions.
Bottom line, every single post requires approximately 10 different prompts to refine the extraction.
JobLens is cool! This past month we had the same idea -- it's been a fun project:
https://hnjobs.u-turn.dev
ChatGPT does an incredible job parsing, but then lots of effort goes into normalizing and deduping each field. Long story short, your results look quite good to me!
Product engineering leader helping build great teams. Twenty-plus years of software development. Over the past decade have helped turn around multiple products and teams in crisis.
Currently focusing on helping organizations apply deep learning and LLMs for information extraction from unstructured sources. A recent project in that vain: https://hnjobs.u-turn.dev
When I saw the post about Foundation DB, I remembered the exact same demo running on a cluster of Raspberry Pi instances! Sadly, no memory of it on YouTube.
We use ChatGPT to extract information about jobs and candidates for the past two months. With structured jobs and candidates, you can filter instead of just searching and scrolling.
There's a small write-up on how we went about processing HN Who Is Hiring with ChatGPT:
> Interestingly, those who felt the greatest boost in mood also experienced the biggest drop in heart rate variability.
A drop in HRV isn't generally good. So I check the paper, and found this:
> No significant changes were found in heart rate variability or resting heart rate over the course of the study in either of the groups (Figures 4C and 4D)
There was a reduction respiratory rate for those an increase in daily positive affect. Bottom line, unclear if this particular study points to a positive health outcome other than feeling happier.
I don't think it's analogous to gittip where someone is attempting to send funds to a particular repository or organization, and, I assume, you don't know the status of your funds.
When subscribing to StackAid, you indicate that you'd like to support your dependencies. It's hopefully clear that their dependencies have to participate to receive their allocation. We're also upfront that not all OSS projects are interested or currently have a relationship with StackAid. As a result, allocation amongst dependencies will shift depending on the OSS project interested in participating, and that's an important feature that sets it apart from other funding models.
All of that said, we've tried to be clear in the messaging and setting expectations. We're very much open to feedback to improve what we communicate.
We support Java through a GitHub Action that generates a stackaid.json file that we consume with your first and second order dependencies. More details here:
https://www.stackaid.us/#stackaid-json
I wasn't aware of SPI. I'd be happy to look into it and reach out to them.
To give some context, if SPI were interested in receiving money for repositories associated with SPI, they would work with the developers to claim those projects on StackAid and associate their Stripe account. SPI's governance model can then decide how to allocate those funds.
We thought a lot about the fairness of allocation. Large funding for tiny packages is a clear edge case of the StackAid model.
We are thinking through how to give subscriptions control over projects and organizations to exclude, so you could choose to exclude tiny dependencies in your subscription.
That said, defaults matter, and so it's still worth considering the implications of small projects getting a large amount of funding:
1. Subscriptions/open source repositories might be ok with a rimraf getting a lot of money, especially if it funds those developers to build other things that are high impact.
2. It might also inspire other open source projects to potentially compete for those funds in terms of offering something more or encouraging tooling that incorporates that functionality, thus creating a smaller set of dependencies.
Re: Electron getting much less than these smaller projects in the simulation.
Electron getting less money is a function of the NPM packages we discovered and sampled. That could or could not be a representative set. You could argue that Electron shows up more in non-public repositories. It doesn't mean that Electron would get more money than rimraf, but the gap could be significantly less.
No money goes to a repository that doesn't participate in StackAid. Whatever is allocated to them from a subscription, gets reallocated after a certain period of time.
We are busy thinking through the best way to provide controls around allocation including specifying repositories/organizations you don't want to fund. This feedback is what we wanted to hear before we embarked on an approach.
Some organizations might also specify that they don't ever want to receive money, and allowing us to avoid waiting to reallocate the money intended for their repositories. This is another option we're planning on exposing.
I'm the person who did the original prototype in Python/Pandas. The purpose of the prototype was to prove that with the right data presentation, you could process more data and cut processing and storage by over two orders of magnitude. I picked search rankings given the size of the data and the struggle the legacy system had in terms of showing more than a limited amount of data.
Previously, the MySQL solution had many rows for each ranking spanning several tables with all kinds of other data not related to showing the aggregate statistics that were relevant to Moz's users.
The solution was processing the raw data in batch and applying categorization to have an integer representation for all values. These changes led to a very compact representation that could quickly be loaded into memory and then filtered/aggregated.
Storing the results as a CSV wasn't important. It just turned out that having the static data allowed me to effortlessly scale-out serving since the data was only updated once a day or once a week, and it was append-only.
How big of a difference did it make? All of the CSVs individually compressed was less than 20GB. The production system served all user data off of a ~60 node MySQL cluster, with rankings being the most costly in terms of processing and 2nd in terms of disk usage (from what I remember).
Also, Pandas was blazing fast at loading several megabytes of CSV data (< 80ms @ the time). If I had to do it again today, I'd probably use Apache Parquet instead.
The most important insight that carried through the various solutions was to pre-process the data so that it was easily consumable for the task at hand. The languages (Python/Elixir) didn't make a difference, in my opinion. That said Pandas is fantastic, it made working with that data in-memory very easy, at least for this prototype.
This is super helpful. Agreed with other comments about making this available more readily on the DO Managed DB pages.
Having looked through the set of extensions, wal2json (https://github.com/eulerto/wal2json). That makes sense since there's no way at the moment to get access to replication.
However, the use case for wal2json in addition to the access to replication is that it's the best way that I know of to get reliable change notifications from the database. I personally use this to pipe the JSON blobs into message queues to be consumed by applications.
As a short aside, I don't use PostgreSQL's NOTIFY/LISTEN because, as I understand it, you will not receive messages if there are no connections to the DB -- this is a show stopper for me.
AWS RDS allows this and Google's PostgreSQL does not. I've personally abandoned Google's version first partly because it seems to have stalled in terms of upgrades, and AWS for being way too expensive for what you get. Although it doesn't work for me at the moment, I'm hoping the DO option will work for me in this regard in the not so distant future.