Really awesome seeing more people work on this! I’m one of the founders of Opik https://github.com/comet-ml/opik which does similar things but also has a UI and supports massive scale. Curious to hear if you have any feedback!
Excited to see more people building in this space. From what we've seen with customers it's critical to be able to compare what you're seeing in production to what you trained on (rather than historical period). That's almost the textbook definition of drift. Do you have a sense on how to approach that?
At Comet.com (disclaimer: i'm the CEO/Co-founder) we provide experiment tracking and artifacts management so we have the training distributions for comparison. I'm always curious how it looks like for a monitoring only solution
Hi HN! Excited to share with you our recent open source project CodeCarbon - a collaboration between Mila, BCG GAMMA, Haverford College, and Comet.ml. CodeCarbon records the amount of power being used by the underlying infrastructure from major cloud providers and privately hosted on-premises data centers. Based on publicly available data sources, it estimates the amount of CO₂ emissions produced by referring to the carbon intensity from the energy mix of the electric grid to which the hardware is connected.
Hey! I'm one of the founders at Comet.ml. We believe that Git should continue to be the approach for managing code (similar to dvc) but we adapted it to the ML workflow. Our approach is to compute a git patch on every run so later you can 'git apply' if you'd like (https://www.comet.ml/docs/user-interface/#the-reproduce-butt...).
Streamlit is awesome but solves a different use case. There's definitely similarities but also a few core differences. 1. Comet Panels reads the experimentation data from our API/Database. Streamlit is much more like Jupyter notebooks where you'd have to setup your own data store and implement logic to update it with new experiment data.
This also means that if you pull a visualization from the gallery it should work out of the box as the API is standard among everyone!
2. Comet Panels is freely hosted so no need to setup a webserver
3. Comet Panels is natively written in JS/HTML/CSS so you can use any JS library out of the box. Streamlit is written in Python and you might be limited to the built in UI components (or you can build a custom on which is also in JS/HTML/CSS)
Hi HN! Really excited to share this with with those of you training ML models. Comet Panels is an easy and free way to build visualizations, widgets and apps in HTML/JS/CSS on top of your ML experiment data and a gallery where you can share your work with the world.
Panels is built on our experiment management platform (100% free for individuals). Essentially as a user you can report all of your metrics, params, assets, data samples etc. Everything stored is also accessible via our JS SDK so it's really easy to fetch data for your panels.
The main idea is that visualizations aren't stale. Every time a new experiments is reported your visualization will update with that additional data.
Hi M0zg! Gideon from Comet- sorry to hear you're having issues. Did you every try to report these? if you share more info at [email protected] or at our slack channel i'm sure we can fix it / improve. On a general note: 1. you can see min/max values in the metrics tab for finished/running experiments. 2. we spend tons of time on performance but these are actually difficult problems to solve, i.e if you have ten charts all showing 10k data points all updating in real time. That said if you share your project we can use it to improve. Finally the SDK is designed to never crash or slow down your training and this is the first time we've heard that complaint - again please ping us so we can figure out what's going on.
Really excited to see this relaunched. Every DS team has issues around dataset management. We previously shared a tutorial on how to get a fully reproducible pipeline with Quilt + Comet.ml https://blog.quiltdata.com/building-a-fully-reproducible-mac...
About us: Comet is doing for ML what Github did for code. We allow data science teams to automatically track their datasets, code changes, experimentation history, and production models — creating efficiency, transparency, and reproducibility. Ever wondered how Linus felt when he invented Git? How about the brains behind JIRA?
Join us to help write the future of ML workflows and teams.
A common error is target leaking. An AutoML system will likely consider this a "strong feature". This is where having someone that actually understands the business domain is critical.
There's no question that there's value in AutoML system yet most ML production systems I've worked on / seen were way more complex than feature vector -> model -> prediction.
You likely have multiple models, pipelines, normalizations and plain old conditionals. Hard to automate all of this.
Comet.ml - multiple engineering positions | New York City | Full-time | Onsite (NYC) or REMOTE
www.comet.ml
About us:
Comet is doing for ML what Github did for code. We allow data science teams to automatically track their datasets, code changes, experimentation history, and production models — creating efficiency, transparency, and reproducibility. Ever wondered how Linus felt when he invented Git? How about the brains behind JIRA?
Join us to help write the future of ML workflows and teams.
Sure you can reuse tools to achieve similar results. As with everything else the devil is in the details. Does your monitoring system saves results forever or it only let you report 90 days back? Can you compare two runs in a meaningful way? i.e not just logs but also interactively plotting exploring your results? Do you need to spend hours to instrument your code? Can you sort jenkins job by a parameter/metric? What about reporting new results to an existing experiment? There's many more examples. But in any case if you can reuse your CI/CD system for ML experiment management you should do that. Another question worth considering is that if this is a "failed idea" why would engineering led tech companies build these systems? Obviously they tried reusing their current tooling.
The tools we've been building for the past fifty years were designed for software engineering. Machine learning workflows are different in many ways and as such require new tools and approachs. That's at least our perspective.
Runs and experiments are not 1:1 mapped. A single container run could generate multiple experiments such as with the case of parameter search. Additionaly traditional tools for version control are not well suited for ML results and exploration. That said code is still a big piece of the puzzle. Our approach at Comet.ml is to snapshot everything whether it runs on a container or not and tie that back to git.
Comet.ml | ONSITE | New York, NY | Senior Backend Engineer | Full-time | VISA | Comet.ml is doing for Machine Learning what GitHub did for code. We allow data science teams to automagically track their datasets, code changes, experimentation history and production models creating efficiency, transparency, and reproducibility.
Ever wondered how Linus felt when he invented GIT? How about the brains behind JIRA? Machine Learning teams operate like software teams 15 years ago and we started Comet.ml to write the rules of ML workflows.
We're looking for a senior backend engineer to join our team of 8. Comet.ml is built using microservices (Java/Python), Redis, Kafka, MySQL, NGINX. We are funded by Amazon, TwoSigma, Trilogy and others.
My biggest issue with notebooks is that we're throwing aways years of best practices. Notebooks often lead to untested code with poor structure, frequent use of global variables and readability issues. Even using it as debugger is limited since there's no way to step into functions.
I was mostly referring to coupling code with results. For example you have a code that loads a dataset from S3 and then trains a neural network. If you only use git you're likely to lose the hyperparms info (which is often passed as command line arguments) and your metrics/results.