Knuth is also still reviewing TAOCP-related email and issuing reward checks. A teammate received a reward check for 1 hex dollar last month after finding an error in Seminumerical Algorithms. Along with the check, there was a printout of the original email with hand-written annotations.
SheetJS | https://sheetjs.com/ | Software Developer | Full time, Remote (US) | $165K - $240K
We're a bootstrapped company building open source solutions for spreadsheets and structured data. With over 1.8M unique monthly visitors, companies across the business world turn to us for challenging data processing problems. Over the last 10 years, we have pushed the boundaries of JavaScript and the web.
In this role, you will master new and established technologies while working on high-impact projects used by millions of people across the world. Balancing research and engineering, you will design and implement creative solutions that draw from your academic and professional experience.
SheetJS | https://sheetjs.com/ | Software Developer | Full time, Remote (US) | $165K - $240K
We're a bootstrapped company building open source solutions for spreadsheets and structured data. With over 1.5M unique monthly visitors, companies across the business world turn to us for challenging data processing problems. Over the last 10 years, we have pushed the boundaries of JavaScript and the web.
In this role, you will master new and established technologies while working on high-impact projects used by millions of people across the world. Balancing research and engineering, you will design and implement creative solutions that draw from your academic and professional experience.
Under the hood, numpy and friends punt to native code. FFI / native module performance becomes a bottleneck, and Bun is correctly focusing in this area. Shumai [1] is a cool library for Bun that also punts to native code
JavaScript has a massive and unrivaled ecosystem outside of core data analysis. The main hurdle to adoption is ecosystem and community. A generation of applied mathematicians and data scientists were trained on Python. Python overcame older ecosystems like Matlab and Fortran for its ease of use for other general computing tasks (Matlab is a decent language for large array manipulation but soft tasks like string processing are kludgy compared to Python).
In the same way that Python developers are interested in extending their ecosystem to other spaces, JavaScript developers are interested in growing into the data science space. But that doesn't happen unless there's a clear benefit for switching.
What advantages do a native JS solution have over Python? Performance: There are arguably many more developers and companies focused on pure JS performance than python. Community: A JS solution opens up a much larger pool of talent to build innovative solutions. Cost: Running heavy computations client-side, in the web browser, reduces costs for service providers.
The hardest unsolved problem is document compatibility. It's not enough to have a cool tool -- you need to be able to interchange data with existing users of Excel and other workflows. This was an important piece during Excel's journey to overtake Lotus 1-2-3.
We (https://sheetjs.com/) have been looking into document compatibility for 10 years (celebrating our 10 year birthday this week!), and our eponymous open source project https://github.com/sheetjs/sheetjs is used by companies large and small. It's not a particularly glamorous subject and doesn't tend to electrify people in the same way as build tools or frameworks.
.
There are ways to improve upon the space, but the problem is that the world has changed. Excel was designed to be the "center of the universe", a creative substrate that bypassed org security policies and enabled extremely flexible line of business tools. Excel was designed to be used by one user at a time, with fundamental inconsistencies blocked at the UI level (for example, try entering a bad custom number format). This made sense 20 years ago, but it doesn't make as much sense now.
The current crop of SaaS companies effectively monetize access to the data. They aren't incentivized to make it easy to export metadata back to Excel -- they want to keep you using the software. This runs at odds with the data portability and freedom that is needed for a successful replacement.
Whatever will replace Excel won't be a facsimile of the current tooling (what Google and Apple are trying to do), nor will it be a siloed experience (what the myriad of SaaS apps are trying to do).
Unfortunately the source spreadsheet was not shared, but the root cause was a misreported codepage (which you can actually fix with a hex editor!)
"Â ", where the second character is a unicode non-breaking space characters (U+00A0), is the CP1252 encoding of the byte sequence [0xC2, 0xA0] which is the UTF-8 encoding of "\u00A0". So all that really needs to happen is to get Excel to think that the file is UTF-8 encoded.
The fix? For XLS, there is a special Codepage record that indicates how strings are to be interpreted. The magic value to force UTF-8 interpretation is 65001, and it is possible to use a hex editor to find exactly where that record is located and change it.
This password specifically refers to "Password to Modify" and Excel 2019 / 365 clearly warn in the reenter password popup:
> Caution: Password to modify is not a security feature. ... Malicious users can edit the file and remove the password.
This type of "protection" is also present in the VBA blobs (where you can change a few bytes and work around the VBA protection)
Saving a file with a password to open actually employs encryption. The exact setting can be tuned with registry settings, but is typically AES-128-CBC.
Saving a file as "read-only" encrypts the file with the standard password "VelvetSweatshop"
Hi, random developer that somehow ended up in the top 500 but didn't have 2FA turned on (https://www.npmjs.com/package/xlsx)! npm inc invalidated all of our authentication tokens in mid April and we have been unable to sign in via the web interface since then. Assumably the same fate befalls other random devs.
Software Arts published an official technical specification of the DIF file format in 1983. [1] (to save a file in the DIF format, /S#S in VC)
It is certainly an interesting format, and Excel completely mangles the data (whether it was intentional is up for debate). For example, the value
1,0
"0.3"
represents the literal string 0.3 in VisiCalc (the 1,0 indicates that the next line is a string). Excel ignores the typing and tries to parse as a value, interpreting it as the number 0.3. Like with CSV, there's an awkward formula workaround for generating a file with the text "0.3":
Maybe a story from a maintainer would help. To contextualize, the main SheetJS open source project https://github.com/SheetJS/sheetjs has over 28K stars.
tl;dr: the project involves "crowdsourced research" which benefits from popularity.
The main social goal with the project is data preservation and integrity. Large-scale economic and political decisions are made from data and analyses in spreadsheets. For example, last year in the UK, COVID cases were underreported thanks to Excel minutiae https://www.bbc.com/news/technology-54423988
Due to various corporate stratagems, the older data representations were intentionally obfuscated. To support Excel, many developers poked around at Excel files and guessed at the structures.
In this environment, the biggest challenge is finding worksheets with random corner cases. These types of files are not easy to create and fuzzing has limited effectiveness. This is where open source and popularity come into play. The open source and JS nature of the project helps reduce testing friction (https://oss.sheetjs.com/ runs in the web browser, no need to install anything) and encourage bug reports with test cases.
There will always be "entitled users" and "low quality bug reports" but that comes with the territory. There are also meaningful issues and code contributions. Efforts at trying to prevent the low quality contributions also discourage higher quality contributions.
Pre-Unicode issues still haunt us today, kept alive by various file formats that rely on system encoding.
Under the Apple "Mac-Roman" encoding [1], the standard MacOS encoding before OSX switched to Unicode, byte 0xBD currently is capital omega (U+03A9 Ω). However, in the original 1994 release of the character set, they erroneously mapped to the ohm sign (U+2126 Ω) Apple eventually fixed this in 1997, as noted in the changelog:
# n04 1997-Dec-01 Update to match internal utom<n3>, ufrm<n22>:
# Change standard mapping for 0xBD from U+2126
# to its canonical decomposition, U+03A9.
However, in 1996, Microsoft copied over the mac encoding to CP10000 using the incorrect character [2]. Unfortunately the codepage was not corrected when Apple realized their mistake.
This discrepancy leads to a huge number of strange issues with various versions of Excel for Mac (BOM-less CSV, SYLK and other plaintext formats default to system encoding) and other software that use Microsoft's interpretation of Apple's Mac-Roman encoding rather than Apple's official character set mapping.
Our story is very similar. I wrote a small library for converting XLSX and XLS files to CSV. Over the years, that grew into one of the most popular open source libraries on npm/github: https://github.com/SheetJS/sheetjs
Excel has to contend with nearly 40 years of backwards compatibility (MultiPlan, the predecessor to Excel, was released in 1982) and a deep userbase that literally has decades of experience and muscle memory with the software. The Symbolic Link "SYLK" file format introduced in MultiPlan is still supported in recent versions of Excel, leading to the infamous CSV "ID" issue.
Many of our users still run very old versions of Excel and Windows (e.g. Excel 5.0 on Windows 95) because a change in a future version of Excel caused problems or gave different results.
When saving as CSV, Excel will use the regional "List separator" setting. You can override this in Windows 7 with Region and Language > Additional Settings > List separator.
If you are trying to generate a file that plays nice with Excel, there is a way to force a specific delimiter with the "sep" pragma:
We have a few customers in reinsurance, and for the most part the goal is to do the opposite of what the python solutions try to do. Instead of integrating foreign stuff into existing workbooks, the goal is to retain the existing worksheets as source of truth and build modern tools around the files. The most common use case is building out a web interface to replicate the Excel formula engine.
In the python space, there are libraries like openpyxl and xlrd, but the real hurdle is introducing python into an ecosystem which otherwise has no natural knowledge. JavaScript is the language of choice for modern Excel addins as Excel provides an actual API for it https://docs.microsoft.com/en-us/office/dev/add-ins/referenc...
