Kx released a 32bit, free-for-commercial-use version in 2014 and then reversed course around a year later after banks and hedge funds surprised them by flocking to it for a large subset of their developers / dev machines.
Hopefully they stick to it this time around. It's an incredible system. It's the only thing I've ever used, including Pandas, dplyr & Matlab, where someone could stand over my shoulder asking data analysis questions and I could answer them, on the fly.
LLM's, though notoriously bad (so far) at KDB+/Q compared to other languages, are still a godsend for folks getting started. I recently returned to writing Q after being away from it for 8 years and I've been amazed how good even Google's AI suggestions have been at helping with functions & queries.
Getting started tip: try using Q strictly as a query language, avoid K. Do everything else (data shoveling, devops,...) with a different language.
I'm always stunned to see the level of VB6 expertise that remains in the world even two decades after Microsoft left it for dead.
Just look at the effort and knowhow that went into this VBA function that resolves the local file system path from the https url of workbooks synced to OneDrive/SharePoint:
I've been surprised to see many pro devs using Excel/VBA as a secondary tool.
One example: a couple years ago I was working with a big hedge fund and one of their data analysts sent me an Excel model he had built and I was tickled to see the .xlsm extension (i.e., VBA code on board).
"Ahh ha", I thought, "Let's see what these macro-recording cowboys have been up to."
There was a lot of VBA inside, all written by this Caltech comp sci data analyst who was a Python superstar. The VBA was for pulling data from a database, putting it on a sheet, building some formulas, and some pretty formatting. There were even a few userforms!
I teased him, "VBA? What else are you guys using over there? A cotton gin and a steam shovel?"
I was startled to hear him heap praise upon Excel and VBA instead of the usual complaints.
He said something that stuck with me, "Excel makes it easy to understand the dependency structure that is implied by computations. If I had done this in Python, I'd be answering questions about it all day long."
He has a younger sister. As for other outlets for making friends and socializing, team sports are indispensable. Also great, and surprisingly underutilized -- in my observation -- are playgrounds.
With playgrounds you get the best part of school, recess, without the rest of it.
If parents would look up from their phones for a minute and watch what's going on in playgrounds they'd be amazed.
Recently I saw about 20 kids, across a wide age range, hold a spontaneous, well-organized game of Among Us.
You won't just make friends there; you'll run in Draco Malfoys too; and that's also valuable.
We started homeschooling by happenstance when it turned out that kindergarten was going to conflict with our son's beloved jiu-jitsu class. The class was a bit of a drive and we weren't going to be able to get there in time for his favorite part: the 30 minutes of horsing around on the mats before class started.
We thought, "Well, it's only kindergarten". It worked out well enough (knock on wood) that we just kept going.
Compared to my own harried "THE BUS IS COMING!!!" sleep-deprived school days, it has been a wonderful change.
If I could go back in time, I'd put my mom in an ankle lock until she agreed to do the same for me.
Especially since I learned that in my home state homeschoolers can participate in high-school sports. In high school I had to wake up at 4AM for 5AM hockey practice. If I could have returned home afterwards and slept, rather than going straight to school, I would have thanked the Gods.
I agree that insider trading is an unlikely explanation for the reason you mention.
It could be that an early algorithmic advantage compounded into an incumbency advantage: better research tools; better data; better pay for better scientists.
The 'secret sauce' may have started out as novel discoveries and, over time, become 'just being RenTec'.
I have been consulting at a world-renown hedge fund that has a lot of F# including nightly processes that have apparently been running for years with zero maintenance. You'll never hear them talk about it or see them at meet-ups though. The CTO is a bit of a maverick -- I'm not suggesting that this type of quiet adoption is common.
If you use collection classes in vba you can run into this bug pretty quickly. Last year I deployed an Excel add-in to a client that enabled them to run Snowflake queries concurrently from vba and this line:
If queries.FilterHasError().Count > 0 Then
was being evaluated as true even when the count was zero.
I had developed the add-in using 32-bit Excel. This was an extremely confounding troubleshooting experience. Eventually Google & Stackoverflow rescued me from believing I was going crazy.
This debate about whether to use named-ranges in Excel formulas brings back some distant memories. I was on a team at Goldman Sachs that built Excel tools for the investment bankers and because of that we'd help out with the new banker Excel training held at the giant Chelsea Piers sports complex.
Every year freshly minted MBA's would begin the training and immediately become aghast when the trainers told them to never, ever, use named-ranges in formulas. Not only had the trainees been explicitly taught to use named-ranged in their MBA programs, but any idiot could plainly see that [=enterprise_value/ebitda] was better in every way than [=C13/F22]. More expressive, more readable, easier to spot errors, etc.
The trainers would argue that in an MBA program you build your models, submit them, and move on to the next assignment, so you don't get a view of the longer-term problems that arise from named-ranges. What does [=ebitda] actually tell you? Is that the last quarter's actual number, is it the current estimate of the next fiscal quarter, is it a 12-month blended forward estimate?
And as you try to solve those questions with [=ebitda_est_next_fiscal_qtr] you often end up with two more problems: an unambiguous name for you can unambiguously mean something else to someone else and as you update your model over time, if you forget to update your named-range references, you have formulas that look right, but are wrong, e.g., [=ebitda_est_next_fiscal_qtr] now refers to an actual reported ebitda number, not next quarter's estimate.
[=C47] while not telling you much at all, is unambiguous. It doesn't look "right" or "wrong". It can't mislead you. If you want to know what it is, go look at cell C47. The new MBA's would argue, "Wait a second, C47 could now point to the wrong data as well, and it could be mislabeled with a stale row or column header. And the trainers would reply, "Exactly, and when you go to see what C47 represents you will have the context to recognize those errors and fix them."
I encountered exactly this when my brother went to film school. The students would rotate roles and it was his turn to be the producer. One day I came home from work and he was having problems with his Excel-based budget sheet. I noticed that his row of sums at the bottom of one sheet had hard-coded values rather than formulas. Thinking he had inadvertently pasted over them, I told him, "Look, you killed your formulas." "What are formulas?", he asked. Turns out that he had been tallying up the columns with a calculator and then typing those totals into Excel.
Same for me. My high school and college hockey coaches could really let you have it. They never pulled you aside and did it in private either. My college coach had episodes that would make even Bobby Knight look like a pussycat. He once had a roll-on-the-floor grappling fight with a teammate in the locker room between periods. (Coach had a big tactical advantage: he wasn't wearing skates) When I ended up working on a trading desk the impromptu performance review broadsides -- in front of everyone -- felt very familiar.
More color on the original FoxPro matching engine:
"Levine found a simple but hugely consequential 'trick' to speed matching. In Levine’s later paraphrase, what the 'enter2order' procedure did when the Island system received a new order was to: 'See if there was a record from a recently cancelled order that we can reuse for this new order. This is hugely important because that record will likely still be in the cache [fast internal memory] and using it will be much faster than making a new one. (Levine, n.d.)'"
I agree on the key verticals approach as being the best bet. Perhaps the best way to dip your cup into the raging river of Bloomberg cash flow is to take a page out of the Stripe/Twilio book and focus on API first, with phenomenal Excel and Python interfaces, docs and examples. Just forget about a terminal-like UI, or any UI at all and build the tools that make it easier for other people to build their own niche apps. Money.net, Sentieo, YCharts, Koyfin -- my observation is that these all treat the programmability of their data and analytics as a secondary focus, or not a focus at all. It would be nice to see someone come along and do it the other way round.
I've been waiting 15 years for litigation in this arena. If prices, like player statistics from an NHL game, are historical facts which can not be copyrighted, why don't we have better and cheaper public market data API's by now? How have the exchanges been able to maintain a stranglehold over distribution of historical facts?
Speaking of this breadth, a few years ago I looked at Bloomberg's PORT suite of portfolio performance attribution and risk management functions to see what they offered and how programmable it was. On top of the usual extensive help docs, they had 56 white papers totaling ~1400 pages (as of June 2017) explaining the calculations.
Just like you say, I passed the AP calculus exam in high school and then when I arrived at college my advisor, who also happened to teach a freshman Calculus 1 section, told me, "I know you passed out of it, but why don't you sit in on a couple of my classes and see what you think?" I ended up staying the whole semester when it became clear that I didn't understand anything about Calculus.
The desk-aligned dev spot is a tough one to land. If you’re not hailing from CalTech, MIT, et al, I suggest the highest-probability strategy is an indirect one: get a client-facing job at hedge fund administrator, get great at Excel+vba; over-deliver for the fund’s CFO and/or traders in all their post-trade questions (that’s why Excel+vba expertise is important: it’s easy to deploy to them as just an Excel file); get recruited. The right spot at a fund administrator is much less competitive, and you get an opportunity to impress the same people you would if you were hired directly onto a desk.
Excel+vba is an odd beast because, nowadays, it's rarely seen as a differentiating skill -- it's not taken seriously -- and yet on almost any trading desk there is an infinite series of todo's where Excel+vba is just the right tool for the job.
One thing to consider, which I've found powerful, is a hybrid approach where you use vba for what it's best at -- interacting with the Excel object model -- and have the vba load and call a C# xll that leverages the .Net libraries and does all the work that would be tedious to do in vba. For your users, they would simply open a "host" .xlam file, which is the only one they would be aware of, and this file would load additional .xlam or xll files and typically also check for updates to these files. No installer necessary.
For example, your vba code would pick up a range of index (or ETF) tickers from a sheet, use Application.Run("lib.constituents") to call a C# function that pulls the stock tickers and weights that represent each indices' constituents, your vba code would receive back an array of arrays and use Excel's row grouping feature to group the members & weights under the parent index and perhaps add some formatting and\or formulas.
The advantage of this division of labor is that you get faster dev & debug feedback loops within your Excel-centric code, while minimizing the amount of time you spend working in a dev environment that has stood still for 20+ years as others have raced far past.
Your Excel+vba application’s features remind me of when I joined an investment bank in 2000. I had come from an insurance company where I was considered the Excel\vba wizard, and I was impressed, in the extreme, by my new colleagues’ approach to Excel development. Even during the interview process I had realized that, when it came to vba, I was but a babe in the woods. They had auto-updating code (when you “published” code, all client workbooks downloaded the latest version), code-generated collection classes, interface inheritance, tests, error detection, higher-level functions via Application.Run(), self-contained worksheets with embedded vba code that would operate even if moved to a new workbook.
Inevitably, new hires would be unhappy that they did, in fact, have to write vba code all day and would argue for switching to a better language. Our team manager would say, "Vba gives us a superpower no other language does: we can deploy whatever we want, whenever we want, to whomever we want. In any other language, getting 'Hello World' in front of a user is a six month project."
Good point. I've seen miracles occur simply by refactoring an Excel workbook such that the inputs, outputs, and calculations to map the inputs to the outputs were all delineated into their own clearly-formatted bounded context. And also, when you do drop into vba code, simply organizing the code so that interactions with the sheet are easier to understand, and isolated -- and focusing on "seriality" (no feedback loops, minimize code triggered by events) can transmogrify vba code from the Bogeyman into a friendly neighbor.
Hopefully they stick to it this time around. It's an incredible system. It's the only thing I've ever used, including Pandas, dplyr & Matlab, where someone could stand over my shoulder asking data analysis questions and I could answer them, on the fly.
LLM's, though notoriously bad (so far) at KDB+/Q compared to other languages, are still a godsend for folks getting started. I recently returned to writing Q after being away from it for 8 years and I've been amazed how good even Google's AI suggestions have been at helping with functions & queries.
Getting started tip: try using Q strictly as a query language, avoid K. Do everything else (data shoveling, devops,...) with a different language.