Hi, this is Simon, one of the team members that worked on labeling.
I'm not sure I understand what you mean by building feature sets.
Regarding workflow, we are still open to suggestions, but the general concept is treating annotating a bunch of files as a feature.
Hello HN! DVC (dvc.org) and MLflow (mlflow.org) are two open-source projects that are very widely adopted, each for its own specialty. DVC excels at data versioning, and MLflow is used for many things (it's actually multiple tools combined into one), but mainly for its experiment tracking and artifacts logging capabilities. Both tools are open-source, but this doesn't mean that you can easily have an open-source project that uses them. You cannot share the data you versioned using DVC with the open world, and you cannot display the experiment page MLflow generated for you to everyone.
I joined DAGsHub a year ago, and I really loved the idea that MLOps tooling should also be made part of the open-source workflow. A few months back we worked on letting anyone with a repository use DVC to version large files that could be pulled by anyone. Over the path month, I've been working on an integration with MLflow tracking, meaning that your experiments are now a part of your project, and when you log metrics, they are visible by anyone that has access to the code. The permission dynamics make sense - If you can read the code, you can see the experiments. If you can push code, then you can log metrics as well, etc.
I think this is very cool, and it comes with a few bonuses:
1. Better UI for comparison - a repository can host multiple MLflow "experiments" and all the runs are comparable across experiments. This is not yet possible with the vanilla MLflow UI.
2. Integrating with DVC - If you work with both systems you can have real-time experiments with MLflow but then save your models and metrics as part of the next commit and see both of them in the same experiment table.
We're still not supporting MLflow artifacts, but this will probably come in the near future, depending on how necessary this is for the MLflow community.
I'd love to hear your thoughts
1. Create a branch from the current state
2. Create an annotation project on that branch
3. Annotate a bunch
4. Commit to the branch
5. Repeat 3 and 4 like you do with code
6. Open a PR