Amazing work! Reminded me of LLM Visualization (https://bbycroft.net/llm) except this is a lot easier to wrap my head around and that I can actually run the training loops, which makes sense given the simplicity of the original microgpt.
To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog:
> Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37.
to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.
> Consequently, knowledge bottlenecks are a form of operational inefficiency in startups.
This resonates with my startup experience, especially when the organisation is growing rapidly (which is almost always the case for VC-funded startups)
I ended up making my app installable as a PWA based on this conversation myself. Changes were straightfoward to add on top of a Vite + React application: https://github.com/visprex/visprex/pull/62/files
Yes as dimava commented above a PWA would be a great alternative.
> in a way it can be more honest for a developer with any commercial intentions to not open source something while offering paid ways to modify the source code
I appreciate that you made your commercial intentions clear in the comments and I agree that you don't want to walk back your license as it definitely leads to backlash. I think developers (i.e. people on HN) tend to trust open-core models more where there's a clear separation between the core open-source components and a paid offering but I guess you don't have to overindex on those given the general audience of this tool.
I don't know what your ultimate monetization goal is, but in my opinion one realistic path is an one-time only payment of some reasonable amount with the full feature as opposed to subscription (I saw your Stripe link in the source).
Great project! I'm also maintaining a fully in-browser data visualisation/analytics tool [1] and I often found it hard to convince people that their data isn't sent anywhere. It's funny I also say the exact same thing that they can inspect the Network tab. As a potential alternative to open-sourcing the codebase, maybe a desktop version of the same tool (i.e. an Electron app) could help address the privacy concerns if it explicitly states that it won't make network calls, although it's a perception problem rather than a technical one.
Very interesting read, I wish more startups did postmortems like this. It seems like the authors managed to build a set of tools that provided real value that ultimately did not make commercial sense, but would've been interesting to see if this learning:
> Scout was our most-used product. Its users weren’t our target market, but some were. We had vague ambitions to use it as an inbound marketing tool, but we never capitalised on it. This was a missed opportunity.
was applicable in a conservative industry like real estate and government as lots of open-core companies operate on this model (e.g. free open-source software and marked-up hosted solutions).
> That year, I lost about thirty pounds due not having enough money for food. It was in this context that the lab’s principal investigator told me that his funding for my position was running out. Desperate for work, I applied for a job as a web producer and decided to learn to code. My thinking was that it was a stepping stone towards any sort of scientific career, and it would be possible without more formal education or debt. In hindsight, this was the right move, but at the time, it felt like the only move.
This reminded me of the summer when I accidentally landed my first programming gig at my friend's startup as a part-time web developer during university. I was running out of money for food as my research assistant role was unpaid and I asked around for some odd jobs in the city. As a non-CS major, all I had was data analysis skills in Python but my friend thought I was was good enough to help him out with his SaaS startup, and I ended up learning a lot on software engineering principles on that job.
Yes now I'm looking at the tests I can see that the coverage won't increase linearly, but 12% (with some of the hard parts) in two weeks is impressive. I starred the repo, good luck with your journey :)
I will try to add more support for other data formats like xlsx and parquet in the future. Current CSV parsing is also a bit limited (i.e. cannot deal with timestamps) so I will try to update parsers first.
Thanks for commenting this, I would say speeding up the iteration between visualisation steps is the main benefit as you might not want to be thinking about matplotlib syntax when trying to get a sense of data distributions.
> had to rely on tools like Google Sheets or Colab + Pandas for quick cleaning and wrangling before exploring different visualizations.
Yes I had the same experience for analytics work some years ago. As others have pointed out, Visprex only works in a happy path where data is a clean CSV file so will definitely need to work on data cleaning. I have a DuckDB integration planned but not sure if this is easy enough for the target audience. Will try to add some predefinied functionalities, thanks for the feedback!
Yes I do have plans for data preprocessing using DuckDB WebAssembly (I have upcoming features secion in this blog: https://kengoa.github.io/software/2024/11/03/small-software....) but this will require SQL which some of the target audience might not be familiar with. I'm thinking of something like visual query builder from metabase.
> With the UI I want to be able to toggle between different strategies quickly - strip characters from a column to treat it as numeric, if less than 2% or 5% of values have a character, fill na with mean, interpret dates in different formats - drop if the date doesn't parse
Those are really good examples and I can make those predefined preproccesing techniques available as toggles in the dataset tab. Thanks for the feedback!
> how this compares to a spreadsheet CSV import tool such as the one in Excel which is extremely flexible.
I would say the data loading functionality compares very poorly to Excel CSV import for all the reasons you pointed out, and I agree that the users can face those formatting issues which could be resolved in another tool like Excel or Google Spreadsheet for non-technical users and Notepad++ or editors for a bit more technical users. The assumption on CSV files being clean is strong so I will try to surface import errors at least, and in the meantime point to different ways to format the data as those tools will be complementary to Visprex.
> To me, this implies that no steps have been taken to manage user/data privacy.
This is a good point. I fixed the wording and now it simply reads "No tracking or analytics software is used". Thanks!
Added an entry for my data visualisation tool here: https://github.com/stared/interactive-machine-learning-list/....
Edit: found an updated link for seeing theory so I fixed it in the PR above. Feel free to cherry-pick if #24 is not relevant.