Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to people who request but we just wanted to have an email request flow so that we can make sure people know it's just for noncommercial purposes.
Working on a model without this issue. Certainly our goal is totally open models anyone can use for anything.
Yes it's nuanced, but will be simplified going forward.
This uses a fully open source (liberally licensed) model and we also open sourced (liberally licensed) our own training code. However, the uptraining dataset of ~50,000 samples was generated with OpenAI's text-davinci-003 model, and depending on how one interprets their terms, commercial use of the resulting model may violate the OpenAI terms of use. For that reason we are advising only noncommercial use of this model for now.
The next step here is to create a set of uptraining samples that is 100% open. Stay tuned.
The DataFrame is an evolution of the RDD model, where Spark knows explicit schema information. The core Spark RDD API is very generic and assumes nothing about the structure of the user's data. This is powerful, but ultimately the generic nature imposes limits on how much we can optimize.
DataFrames impose just a bit more structure: we assume that you have a tabular schema, named fields with types, etc. Given this assumption, Spark can optimize a lot of internal execution details, and also provide slicker API's to users. It turns out that a huge fraction of Spark workloads fall into this model, especially since we support complex types and nested structures.
Is the core RDD API going anywhere? Nope - not any time soon. Sometimes it really is necessary to drop into that lower level API. But I do anticipate that within a year or two most Spark applications will let DataFrames do the heavy lifting.
In fact, DataFrames and RDDs are completely inter-operable, either can be converted to the other. This means that even if you don't want to use DataFrames you can benefit from all of the cool input/output capabilities they have, even just to create regular old RDDs.
Hey - sorry you had a bad experience. That bug was filed as a "minor" issue with only one user ever reporting it, so it didn't end up high up in our triage. We didn't merge the pull request because it was not correct, however, we can just add our own fix for it if it's affecting users. In the future, if you chime in on a reported JIRA, it will escalate it in our process.
Yeah, you can load Parquet data directly into SchemaRDD's in 1.1 and get the type conversion, including use of nested types. That's the long term solution for all of our storage integration is to go through the SchemaRDD API since it's a standard type description and we expect many data sources to integrate there.
Re: databricks cloud - shoot me an e-mail and I'll see if I can help. Right now demand exceeds supply for us on accounts, but I can try!
The Princeton "University Database" that was hacked is a privately maintained Alumni Association site from the UK. It is not on the Princeton domain or associated with the University at all.
Here's an excerpt:
"Come out and suppoert Chickenshed, an inclusive theatre company based in London that brings people of all ages, backgrounds and abilities together to create groundbreaking and exciting new theatre."
Time will tell how the world will change now that this sensitive information is out in the open.
But surely there are success stories which are outliers/anomalies/red herrings, that you really don't want to try and replicate?
To say that every missed opportunity is a failure is like saying that every CA lottery I didn't buy a ticket for was a mistake. That strategy risks over-fitting to successful fluke's, to people who made bad bets that happened to work out. There is enough loose money floating around today that some people out there are going to make bad bets and succeed.
I don't see why you/YC wouldn't focus on bets where you have an unfair advantage over the market and feel fine passing those up where you don't, even if they have some non-zero probability of success.
Does it always mean that something went wrong if you passed on an eventually successful startup? That seems like an assumption baked into the question and your answer.
At the end of the day you are taking a calculated risk. The interesting question isn't whether you wished you had funded them, because that is asking you to make a risk-free (in hindsight) decision. The real questions is how you decide whether a missed opportunity indicates a lapse in the way you calculated risk/reward, or simply a bet you ended up on the wrong side of despite it being the right bet at the time.
That's the question I'd really be interested in hearing the answer to. How do you decide whether a missed opportunity represents an error in your process?
We'll definitely keep iterating on Dolly and releasing everything openly.