Show HN: SQL Explorer – Open-source reporting tool that Just Works(github.com)
github.com
Show HN: SQL Explorer – Open-source reporting tool that Just Works
https://github.com/explorerhq/django-sql-explorer
22 comments
Nice tool! I built https://sql-workbench.com/ which runs completely in-browser via DuckDB WASM, and enables querying of remote CSV, JSON, Parquet and Arrow data sources, as well as uploaded local files. Charts are supported as well, see the accompanying blog post https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...
I like the tech stack. Does it provide anything like the query catalog in the OP, i.e. the table labeled "All Queries" on the OP demo page?
There‘s a query history tab, not sure if this what you want?
Otherwise, I‘m very open to feedback, just create an issue in the linked GH Repo. Thanks!
Otherwise, I‘m very open to feedback, just create an issue in the linked GH Repo. Thanks!
The idea is that having created a query, chart etc., you can save it for other users to run.
Yes, you can share a query, and also the visualization.
See https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...
See https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...
Is this on GitHub to take a look at the code?
Currently it‘s not open sourced. Eventually considering it for the future though.
Thank you. Look forward to it! How is the data model visuzalization built? Does Duckdb provide that out of the box?
No, it's implemented via parsing the current schema and translating it to a Mermaid ERD, see https://mermaid.js.org/syntax/entityRelationshipDiagram.html
Just wanted to say thank you for all your effort on SQLExplorer, I incorporated it into our open source radiotherapy quality assurance project (https://qatrackplus.com) years ago and it's been a great addition and is used in hospitals around the world :)
That's amazing! Thanks so much for sharing. Happy to chat if there are specific features or functionality that would be useful. Always looking for more feedback.
I spent 10 mins on your docs, website.
Excellent effort overall.
But I didn't know until I read your comment here about the uploading CSV, instant parsing that non technical people may find very interesting. This is something pgweb for example doesn't have.
Your docs are also missing a complete sample env.
See that you've integrated pivottable. Nice touch!
If you can figure out minimal barcharts , you may even have an opensearch/log community interested.
Another killer idea is uploading CSV/json and getting faceted search. No one does this! But maybe distracting to your roadmap.
Keep up the excellent work!
Good luck!
Excellent effort overall.
But I didn't know until I read your comment here about the uploading CSV, instant parsing that non technical people may find very interesting. This is something pgweb for example doesn't have.
Your docs are also missing a complete sample env.
See that you've integrated pivottable. Nice touch!
If you can figure out minimal barcharts , you may even have an opensearch/log community interested.
Another killer idea is uploading CSV/json and getting faceted search. No one does this! But maybe distracting to your roadmap.
Keep up the excellent work!
Good luck!
Thanks so much! Can you expand on the idea -- what do you mean by faceted search? What would that look like?
I think GP is referring to something like excel’s filters, where the UI exposes filter options dynamically based on the data actually available after all existing filters have been applied.
https://docs.datasette.io/en/stable/facets.html
(almost like a group by)
Have you looked at Datasette? It has faceted search by default. https://github.com/datasette
I've been self-hosting this https://github.com/dbgate/dbgate for a few years.
But I like some of the features in SQL Explorer interesting - like Pivot tables and exposing queries as JSON endpoints.
But I like some of the features in SQL Explorer interesting - like Pivot tables and exposing queries as JSON endpoints.
> * All content that resides under the "explorer/ee/" directory of this repository is licensed under the license defined
in "explorer/ee/LICENSE".
Not really open source. If you care about that.
Not really open source. If you care about that.
It’s absolutely open source - and completely free for commercial use. That license simply encumbers that specific functionality from resale. I don’t want anyone selling a SaaS version for profit (if someone wants to do that, they can contact me and we can talk about it).
Nothing wrong with making source-available or open core software.
But saying it's open source when its license does not follow OSI's definition is confusing at best, or misleading at worst.
OSI doesn't have a monopoly on the term, but that's the generally accepted definition (at least IME).
But saying it's open source when its license does not follow OSI's definition is confusing at best, or misleading at worst.
OSI doesn't have a monopoly on the term, but that's the generally accepted definition (at least IME).
Fwiw I thought open source meant the source was available. "FOSS" or "MIT" mean do whatever you want with it
Just discovered this and got the test project up and running... but wondering how to enable CSV import?
The "upload csv file" box does not show up in the test project.
The "upload csv file" box does not show up in the test project.
You got it! You'll need three values set in settings.py:
https://github.com/explorerhq/django-sql-explorer/blob/64170...
https://github.com/explorerhq/django-sql-explorer/blob/64170...
def user_uploads_enabled():
return (EXPLORER_USER_UPLOADS_ENABLED and
EXPLORER_DB_CONNECTIONS_ENABLED and
S3_BUCKET is not None)
Awesome project. But a somewhat irrelevant suggestion. OP could have shared the video via YouTube for better user experience (adaptive bitrate streaming) and also not had to worry about paying for S3.
Thanks for open sourcing, I’ll definitely give this a try!
Any reporting tool is only as good as the data available to it.
I built Multiwoven, a Reverse ETL with SQL capabilities, to sync data from any data warehouse to destinations like this one.
https://github.com/Multiwoven/multiwoven
Any reporting tool is only as good as the data available to it.
I built Multiwoven, a Reverse ETL with SQL capabilities, to sync data from any data warehouse to destinations like this one.
https://github.com/Multiwoven/multiwoven
Great job! And congrats on actually getting something out there. I can really see this being useful for some organizations.
I also envisioned this same type of tool around 10 years ago and it is still on my ever growing list of ideas to implement. I took the idea further to support not only SQL but other languages such as HTML, JavaScript, Python, C#, etc. You could then support returning different types of media based on the URL extension such as .html to return a webpage, .json to return a JSON API, .csv to return a CSV file, etc. As time marched on, many of these same ideas came to fruition in things like AWS Lambda, Jupiter Notebooks, Microsoft Monaco Editor, etc.
I also envisioned this same type of tool around 10 years ago and it is still on my ever growing list of ideas to implement. I took the idea further to support not only SQL but other languages such as HTML, JavaScript, Python, C#, etc. You could then support returning different types of media based on the URL extension such as .html to return a webpage, .json to return a JSON API, .csv to return a CSV file, etc. As time marched on, many of these same ideas came to fruition in things like AWS Lambda, Jupiter Notebooks, Microsoft Monaco Editor, etc.
This is awesome, I hope I get a chance to use it once.
One thought: I think the effort should be put into the UI for the non-technical end users, instead of query builders/developer experience. I would be even fine with a tool doesn't even have a query tool and just executes SQL files from a folder/git repo. The important part would be for me to provide a perfect experience for the end users. Developers usually have a lot of tools at hand to create queries, no need for another one.
One thought: I think the effort should be put into the UI for the non-technical end users, instead of query builders/developer experience. I would be even fine with a tool doesn't even have a query tool and just executes SQL files from a folder/git repo. The important part would be for me to provide a perfect experience for the end users. Developers usually have a lot of tools at hand to create queries, no need for another one.
Yep - that makes sense. The Query pane can be collapsed, effectively hiding the SQL from the end user. This is indeed how a number of people use Explorer. But it could certainly be more optimized, in the direction you suggested. I'll think about how this might be improved!
It was just a thought based on a quick look at the screenshots. I didn't use it yet.
I would use a tool like that as a low-code platform to quickly make data accessible. Might be a different use case than most users are looking for.
I would use a tool like that as a low-code platform to quickly make data accessible. Might be a different use case than most users are looking for.
nice work, it would be better to add graphic walker for interactive data visualization (also include pivot table): https://github.com/Kanaries/graphic-walker
Been thinking Metabase could benefit greatly from AI integration. This kinda does that!
If you're looking for AI in BI, I have something for you: https://www.definite.app/
not open source, lowest cost plan is 1k per month. lol. (for up to 5 million rows ... dealbreaker right off the bat)
I like the simplicity, and yet there is a lot of stuff to do.
I know there are bunch of tools that do this (superset, redash, dbeaver web etc.) but there is a great value in the feature and UX choices of any particular tool.
Keep it up m8.
I know there are bunch of tools that do this (superset, redash, dbeaver web etc.) but there is a great value in the feature and UX choices of any particular tool.
Keep it up m8.
Cheers! Yes - lots of good options, and you said it well.
Great effort! Feels like you should be charging for this, maybe a Pro Plan with SSO (user edits integrated into your history logs) and 1 year support for SMEs to self-host.
I'm genuinely impressed by this. It doesn't surprise me that it took 10 years to make.
loved the demo video! we need more enthusiasm like that, really shows how much you love the work you've put in :)
I'm curious about how the project name was chosen considering there are existing tools with the same name.
Intersting, can you generate fancy printable reports from this tool?
Hmm, depends what you mean by 'fancy'. There is a 'full screen' view of any given report: https://demo.sqlexplorer.io/50/?fullscreen=1&rows=1000&query...
And it can do PDF export (with a plugin).
Would love to hear what you have in mind?
And it can do PDF export (with a plugin).
Would love to hear what you have in mind?
This is really cool!
Hey look, its Redash! https://github.com/getredash/redash
That's a very harsh comment to give someone that has poured hours of their life into something trying to help others unpaid. You could at least politely ask what makes it different than redash.
Perhaps a bit impolite, but no offense taken. It's a very crowded space and there are a ton of good tools! I work on SQL Explorer simply because I get to make the thing that works best for me.
Redash is very focused on visualization. SQL Explorer is not. It is going more in the direction of in-browser analysis.
Redash is very focused on visualization. SQL Explorer is not. It is going more in the direction of in-browser analysis.
Don't worry I was downvoted to oblivion for it!
It wasn't meant snarky but I get that that's what it reads like.
Redash is great and all but if you already have a Django app this is 10 min to have up and running inside that same app
A live demo instance is here (no login or anything required):
https://demo.sqlexplorer.io/
And here's a fairly unprofessional, but very enthusiastic, video tour:
https://sql-explorer.s3.amazonaws.com/Sql+Explorer+5.mp4
The UI is constrained enough that there's very little to learn, while there is still a surprising amount of functionality and flexibility to address a lot of use cases.
Some of the stuff I'm excited about in the latest version:
- Intuitive and obvious integration to ChatGPT / the AI API of your choice. Doesn't purport to be 'magic'. Good prompting + relevant table scheme & data automatically injected into the prompt.
- Create a new connection by uploading a CSV or SQLite DB as a new connection, and it's instantly queryable. CSVs are parsed, types inferred, and a SQLite DB gets created (persisted to s3). - New and improved SQL editor with strong autocomplete (based on your schema), and some fancy keyboard shortcuts.
Some of the old stuff that is still great:
- Pivot tables in-browser, so you don't have to open results in Excel for basic analysis. Unique URLs make everything shareable.
- Expose queries (optionally) as JSON endpoints. Great for prototyping APIs and scripts.
- All of the stuff you'd expect in a reporting tool (email reports, logging, favorites, exporting, etc.)
Hope you enjoy!