I find these retro blog posts from people building LLM solutions super useful for helping me try out new prompting, eval, etc. techniques. Which blog posts have you all found most useful? Would love a link.
I've spent the past few years teaching business analysts at large financial institutions how to automate their Excel workflows with Python. My main takeaway are:
1. Business analysts can spend up to weeks per month updating existing month-end Excel reports with new data.
2. Python automations can reduce the updating process to a matter of minutes.
3. BUT actually understanding the Excel file & writing the Python automation code can take several months.
4. Tools like ChatGPT are hugely useful when given a specific translation task — eg: convert the formula =SUM(A1, B1) to Python — but get completely lost on more complicated translations.
So I'm trying to build an AI-powered tool that can automatically convert Excel workbooks to Python. The key to doing so is 1) using plain old programming to parse the Excel file and build a translation plan (finding all of the unique tables in the Excel file, identifying repeated logic, determining the correct order to output the code), and 2) to prompt an LLM to execute each piece of that translation plan one at a time.
> or that they break from earlier stated requirements as new requirements are requested.
This nails one of the biggest issues that we're trying to help Mito users with -- code that meets __changing__ requirements. ie: the input data column headers change.
The best approach we have right now is giving users flexibility to parametrize their script (ie: tell the script the right column header or reference columns by position instead of name)
It turns out that teams that have moved their data to Snowflake tend to avoid these issues primarily because the schema of their data changes much less frequently.
> Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
I love the idea of giving users feedback on how to get better at prompting the LLM. I think the key to using this approach within Mito is giving users guidance at the right time -- sometimes shorter prompts get the job done, and they're always easier to write :)
A really sweet integration of this approach could be: when the LLM generated code errors or when we notice that the user undoes their previous prompt, we offer the user help in converting non-working prompts into ones that follow best practices of breaking complex tasks down into small steps.
> On the Design of AI-powered Code Assistants for Notebooks - uses Mito as part of their case study
Andrew McNutt, one of the authors presented this paper here: https://www.youtube.com/watch?v=g0prh8mE3bI Their different classifications of notebook code-gen tools has actually been super helpful in my own thinking. Thanks for the help, Andrew if you're a HNer
That's an interesting idea. When converting an Excel workbook to Python, checking your work as you go is the hardest thing for us to convince people to do. (See here: https://blog.trymito.io/automating-spreadsheets-with-python-...). So tools to make that double checking process easier are really interesting to us.
I've personally never seen a Mito users with a detailed enough spec of their report that an LLM would be able to use it to check compliance -- but maybe if we built the functionality users would create them...
Hey, I'm Aaron, co-founder of Mito. Funnily enough, doing "diff detection" in spreadsheets is like the first thing we made when building Mito. We built Git for Excel to enable better collaboration around Excel models -- turns out Excel power users would rather play in single player mode. So it's funny to be exploring spreadsheet difference detection again a few years later. This time, thinking about it purely in single player mode to understand the impact of LLM generated code on your data.
This reminds me of an Upton Sinclair quote: “was one to believe that there was nowhere a god of hogs, to whine this hog personality was precious. To whome these hog squeals and agonies had meaning?”
If when you say GSheets + Airtable, you're looking for a spreadsheet that is designed for data analysis +a let's you name columns and sets formulas to the entire column, then checkout Mito (https://www.trymito.io). Disclosure: I created Mito.
Congrats on the launch Brandon + Brayden. I'm building something similar over at Mito (https://www.trymito.io), although we're Python first instead of SQL first. Would love to hop on a call and learn more!
> In a typical scenario, the customer success team who is in charge of this activity has to work back and forth with the customer. The customer has to resolve manual (unintended) errors.
+ 100. This is even the case when analysts are working with datasets that are created by their colleagues at the same company. Since most companies don't have clear standards for column header labelling, etc. getting a new dataset and incorporating it into an existing workflow requires collaboration with others from inside your company.
Its not an AI based approach, but it is a step up from writing code by hand -- you could try using open source Mito -> https://www.trymito.io -> full disclosure I built it -> to do some of this messy data wrangling. Mito lets you view and manipulate your data in a spreadsheet in Jupyter and it generates the equivalent Python code for each edit. For things like identifying that the data uses '&' and 'and', viewing your data in a spreadsheet is >> just writing code.
Once you generate the code, you could copy it into your pipeline so that you pull the code from the last.fim api, preprocess it with the Python code that Mito generated, and then dump it into the LiteDB.
This is very cool. I'm the creator of Mito [1] -- we're also building a data visualization tool in JupyterLab. The Tabluea approach that you took is really interesting! Going to send you a message -- would love to learn more!
Congrats on the launch! I'm the cofounder of Mito[1], an open-core spreadsheet extension to Jupyter that generates the equivalent Python code every time you edit your data. We also think that combining the intuitiveness of spreadsheet UI with the repeatability and large-data-handling-abilities of Python is going to unlock a bunch of Excel-first analysts to save themselves tons of time by automating repetitive reports.
One difference in our approach is that the Mito spreadsheet goes from Spreadsheet -> Python code, instead of the other way. For every edit you make in the Mito spreadsheet, we generate the equivalent Python code for you.
In practice, this has been really important for us for few reasons:
1) A lot of our users are early in their Python journey. They might've taken a Udemy course or done some Kaggle classes, but generally they are not yet comfortable writing a Python script from scratch. Since they already have a ton of work on their plates, if the option is do the report manually in Excel for 2 hours today or spend the next 2 days writing a Python script to automate their work and save them those 2 hours a day each month going forward, they will probably choose to do it manually. By giving them the Python code for the edits that they make, its more like build the report for 2 hours today in Mito and get the Python script automatically so you don't ever have to build the report again.
2) There are 1 million and 10 things that users want to do in a report, so by giving the user the equivalent Python code, they're able to use the code they've generated as a starting point, not the finish line. For example [2], one really common use case we've seen is Excel workbooks with the following tabs: input_data, Jan 2020, Feb 2020, …, Dec 2022, …. In each case, the month tab corresponds to the same sort of filtering and transformations of the input data. These users get a huge amount of value out of actually having access to the Python code that they generated. The user will use the Mito spreadsheet to generate tab Jan 2020, turn the Mito generated code into a function, and then apply the function to generate Feb 2020 ... December 2022.
The friction of getting started with Mito is something we spend a lot of time focusing on. For example, when it comes to the installation process, not only do users install Mito through a CLI, but because JupyterLab 2, JupyterLab 3, and Jupyter notebooks all support extensions in different ways, there are different installation commands that users need to run to get it working for their specific environment. Initially, we just gave users instructions in our docs about which commands to install for which environment. Now we've built a completely new Python package, the mitoinstaller package, that handles the entire installation process. It downloads Jupyter if they don't have it, detects which version of Jupyter they have installed, runs the installation commands for their JupyterLab version and Jupyter notebooks, and finally starts up the Jupyter server with a tutorial notebook. In the success case, users run two commands and then 2 minutes later have already imported data into their first Mito spreadsheet.
That initial friction reduction is important to our target users, who I would describe in two buckets:
1. Target open source adopters. These users are beginner to intermediate Python users that want to / need to write Python for data analysis. Most of the open source users that adopt Mito are already on their Python journey -- we're not teaching them what Python is or what a notebook is in the vast majority of cases. Many of them have gone through Kaggle courses, taken a couple data science classes at school, or are particularly enginuitive. For those beginner users, and even for people like me who have written pandas code for a few years, some things are just much easier to do in a spreadsheet interface, like creating a pivot table or graph (two of our most popular features)
2. Decision makers at large enterprises responsible for moving their company from Excel to Python. Much like us, these decision makers think a ton about the friction of getting employees started with Python. In most cases, they set up JupyterHub (https://jupyter.org/hub) so users don't need to go through any installation processes themselves, and they control things like version controlling, turning notebooks into reports, etc. They generally also offer/require Python training courses, provide template notebooks, and have data scientists available to help the business end users when they get stuck.
For some non-technical users, Power Query is a better option. If your main purpose is to work with a large data set and then update a PowerBI dashboard, for example, then Power Query sounds like a perfect solution.
But what we see is that there are a bunch of reasons that these non-technical users are excited about Python specifically. Here are two examples:
1. One of the first adopters of Mito, let's call her Shelly, is helping a team of engineers build out a Salesforce dashboard to predict when customers are going to refill an order. Since the engineers don't have the business context to figure out how to make that prediction, its Shelly's job to construct the (in this case pretty simple) algorithm by querying the relevant database, figuring out which fields are accurate (there might be 5 different date fields and figuring out which one is actually when the user last placed an order isn't as easy as it sounds), and then making the prediction for each customer. Shelly then uses the Python code that Mito generates as a communication tool for the engineers. The code is an exact audit log of each transformation she made to her data in order to create the report.
2. Many of the companies that we work with have business specific metrics that they calculate, so they have an engineering team build a Python package that can easily calculate those metrics. Sometimes they will even provide boiler plate Python code snippets to interact with those packages. (In the future they'll be able to import them into Mito!). Its a win for the employees who can rely on the code snippets instead of calculating the metrics manually, and its a win for the engineers who can write Python code instead of M.
The last thing I'll say is that companies are moving to Python because of the openeness and robustness of the Python ecosystem. They are power users of packages like Voila, Plotly. Having employees work in Python opens up a ton of doors for how companies can support them.
Your idea about expressing logic in excel and generating the equivalent code has merit too for a different user base. Let us know when you build it, excited to check it out!
+1, beyond the most obvious reasons that companies are moving away from Excel (too much data to process, not enough robust automation features), there are important workflow management reasons that companies are making the transition.
More and more, we're hearing that companies want to use software engineering practices on their data analytics workflows -- things like version control, easily understanding what edits are applied by looking at the code, and even things like CI to automatically build dashboards from the most up to date data.
While you technically could build tooling around Excel to do a lot of these things, its much easier and already exists in the Python ecosystem.