I have been working on a problem most language detection libraries quietly fail at: short, messy, conversational text. The kind you see in chat apps, support tickets, SMS, and mixed-language messages.
FastLangML is my attempt to fix that.
It is a multi-backend ensemble (FastText, Lingua, langdetect, pyCLD3, and others) with a voting layer built for real-world text. It handles:
Short messages with almost no statistical signal
Code switching like Hinglish or Spanglish
Slang, abbreviations, and emojis
Multi-turn conversations where context matters
Confusable languages like ES vs PT or NO vs DK vs SV
A few design choices:
Context-aware detection so you can pass conversation history and get more stable predictions
A hinting system for slang, abbreviations, and custom rules
Extensible backends so you can plug in your own detectors or voting logic
Optional persistence using Redis or disk for multi-turn conversations
Support for more than 170 languages across the ensemble
Why I built it: most detectors are tuned for long, clean text. They break on "ok", "jaja", "mdr", "brooo", or anything with mixed languages. I needed something that works on real chat data, not idealized text.
I would love feedback from HN on:
How you evaluate language detection quality in production
Whether context-aware detection helps in your workflows
How is the product different from the other test generation tools? How do you check if the are testing the intended behavior. My experience with automated testing solutions has been lukewarm so far.
I did it a little over two years ago. I had a pretty high paying job, low-stress work and desire to make a difference. I definitely miss my fat paycheck but love my freedom of experimentation, learning and doing what I believe in. I learned a lot about building a company from the ground up, but most of the miseries could have been avoided. I had to learn most of this the hard way, which might have been easier when I was working in a big corporation. These are my learnings and may not be right for everyone:
TL'DR: Product is more than just code. It's building something that people love and would pay for. Learn marketing, sales, storytelling. Connect with folks who are not like you.
1. I should have built a network outside of my immediate circle. I am an engineer and my entire work network was engineers.
2. I should have identified some of the best folks in sales and marketing and learned from them. It is easier to know who are the best folks in your company and know them. It's easier when you are inside the company. They can also help you identify/interview other sales/marketing folks in their network when you need to hire them.
3. I should have learned how software products are evaluated, sold and bought in my company. Just a hint, the quality of the product has little to do with whether it will sell.
4. I should have learned the importance of UX design and why certain UX design decisions were made. I should have spent more time talking to designers and learn how to evaluate a good designer.
5. I should have talked with engineers outside my organization. It helps to broaden your horizons and understanding the challenges.
6. I should have understood how business development works and learn how they talk. Most of the sales decisions are made by folks who might not be using the product you are selling.
7. I should have identified who are the M&A folks in my company and understood how the process works. These folks are well connected and usually know others as well.
8. I should have learned storytelling from some of the amazing PM's that I worked with. Some of them were startup founders. I should have tried it internally and improvised.
9. I should have learned from their story, their mistakes and what they could have done differently.
10. I should have learned to talk less and listen more in discussions.
11. I should have spent more time, validating ideas when I had all the time and more money. Some of the groundwork can be done when you are already working as long as you are not building a product that competes with your day job.
12. I should have spent more time reading/writing. You'll spend a lot of time writing emails/ specs in addition to code.
13. I should have taken YC summer school when I was working. I learned a lot when I took it this year.
FastLangML is my attempt to fix that.
It is a multi-backend ensemble (FastText, Lingua, langdetect, pyCLD3, and others) with a voting layer built for real-world text. It handles:
Short messages with almost no statistical signal
Code switching like Hinglish or Spanglish
Slang, abbreviations, and emojis
Multi-turn conversations where context matters
Confusable languages like ES vs PT or NO vs DK vs SV
A few design choices:
Context-aware detection so you can pass conversation history and get more stable predictions
A hinting system for slang, abbreviations, and custom rules
Extensible backends so you can plug in your own detectors or voting logic
Optional persistence using Redis or disk for multi-turn conversations
Support for more than 170 languages across the ensemble
Why I built it: most detectors are tuned for long, clean text. They break on "ok", "jaja", "mdr", "brooo", or anything with mixed languages. I needed something that works on real chat data, not idealized text.
I would love feedback from HN on:
How you evaluate language detection quality in production
Whether context-aware detection helps in your workflows
Ideas for improving code switching accuracy
Additional backends worth integrating
Repo: https://github.com/pnrajan/FastLangML
Happy to share benchmarks, architecture notes, or design tradeoffs if people are interested.