On balance, it will probably be better for you in the long run. I am a high school and college dropout, and nearly as old as dirt. I am entirely self-taught with computers, with no academic credentials. I didn't even take CS courses while in college. I am not a failure career-wise, but I believe that a CS, EE, or similar degree, in hind-sight, would have helped me.
There's 3 main reasons for this:
1) there is a lot of conceptual and algorithm stuff that is drummed into people at school that, while possible to learn on one's own, requires time and effort. If I had time to study Knuth in depth, I would love to do so. This subject matter is very often useful--in hindsight, this is my biggest regret for not having studied CS, because there are gaps in my knowledge that would have been easy to fill had I been studying these things long ago.
2) while employers generally do not require a CS degree, it is very often that their employment screening steps will filter you out if you cannot regurgitate much of the stuff described in step 1: algorithms and coding styles taught in college, the way that colleges teach them. No matter how much PR a company claims for itself in looking for intelligence, creativity, and experience, if you haven't internalized the textbook stuff, then you'll frequently get filtered out of interview process quite early. You'll still be able to get jobs, but just not likely the ones with the biggest marquee names with the highest candidate:opening ratio.
3. If you want to do a startup, you'll want to be in the vicinity of people who also want to do a startup. That's much easier when you're in college, and you get looked upon favorably if you come from a prestigious institution. Seed and angel investors like throwing cash at pairs of kids from good schools who can make something that seems useful and is in a hot space.
So, yes, go to college. Study CS, math and also some business stuff like econ. Avoid humanities like plague.
Also, the main application is in C++. A python script launches the C++ daemons. Perl scripts are quick and dirty tests and deployment scripts. The main hacking I'm looking for is with C++, and I don't care so much if the other stuff gets re-implemented in some other language.
Thank you, Phil. I'm conflicted about this--I was convinced recently to move away from AGPL having to do with what I was previously unaware of as seemingly legitimate acceptance issues. I feel good about using GPL instead of AGPL.
But I'm conflicted about GPL vs Apache (or BSDish) in the sense that I'm getting the message that I have to bend over backwards just a little bit further before somebody, somewhere might be willing to use my software, maybe. Free isn't enough. I also have to let them fork it, keep it proprietary, wrap their own brand around it, before maybe they might consider using it.
That said, I really want people to use it, and of course help me hack on it. But I'm conflicted.
Good thinking, but I think that shifts the issue--namely, that each inter-thread message uses atomic compare and swap to create the message. I assume there'd be a similar bottleneck on the actor that generates the transactionid limited by the number of messages it can send & receive.
Instead, a friend and I have been thinking about how to perhaps modify MVCC to work with distinct transactionid's per partition. Namely, I'm already generating what I call "subtransactionid"'s for each partition involved in a transaction. And those must be ordered for synchronous replication, so I think the way to implement a variation on MVCC may already be mostly there.
I know I still owe you an architectural doc...fixin' ta, ya know.
I've thought about having an actor library, or minimally, to have the actor basis of InfiniSQL independent of specific workload, but haven't thought it through entirely. I'd be supportive of any efforts to that effect if you want to work on it!
1) persistence: battery-backed UPS and synchronous replication. No WAL anywhere. I'm thinking about ways to do disk-based storage without synchronous IO, to provide decent performance with higher storage capacity
2) no joins supported yet. However, the benchmark that I performed (on the blog) involves 3 updates across random nodes. I designed InfiniSQL specifically to perform multi-node transactions very well, because that's the Achilles' heel of every other distributed OLTP system. I plan to implement joins, but expect them to perform decently for the workload you describe.
I assume these shops have Linux in their environments, including the GNU toolchain. There must be some contradiction somewhere that I'm not aware of.
Based on FSF feedback, I'm going to modify the license to include a Classpath-like exception. The intention is to allow people to write stored procedures that link against infinisql without triggering the copyleft. Only if the source to infinisql itself is modified (and distributed) will the copyleft apply.
I'm curious to know the rationale against the GPL in general (not just the AGPL), and how those shops allow Linux & gnu toolchains in spite of their rule against the GPL.
1) I include keystore-like stored procedures in the source. They do get/set with integer key and string val. I haven't done thorough benchmarking, but I expect them to outperform the other benchmark I've published, which is quite a bit more complex workload
2) (camus2) agreed, nothing ever dies in IT. But roll back the clock a few years. How much noSQL would come into exisence if there was a free xzySQL that scaled across nodes, was fast, etc. I believe the answer is that there'd be very few network-based noSQL for operational workloads if that had been the case.
3) jwatte: Yeah! Jagged edges too!
4) stephen24: Also, I intend to change the license from AGPL to GPL next time I push out some code. No excuse not to try it out.
6) diwu1989: Yes and no. Yes, MemSQL is more mature. No,
(a) I'm not sure how MemSQL scales horizontally (especially since that was a feature added after v1 of their code was released), and,
(b) MemSQL isn't free software
7) itsbits: for now InfiniSQL is mainly for hackers and early adopters--the dependencies are pretty clearly documented but it requires some effort to work with in its current state
Those are analytics databases, also known as data warehouses. Optimized for batch reporting. InfiniSQL is geared for operational/transactional (OLTP) kinds of workloads.
Basically, InfiniSQL does not currently support complex indices, so it can't do a TPC-like transaction.
The maximum throughput on 9 nodes is 128,114 per node per second. I don't know if that's 4 or 8 core nodes. If roughly 10% of transactions are multi-node transactions, then 12,811/node/s for multi-node, and 115,303/node/s for single-node transactions.
I don't know if full redundancy for Clustrix was configured, or a hot spare, so I don't know how many actual usable nodes were configured, but likely fewer than 9. But I don't know the precise number.
Roughly 10% of those transactions are contain records on multiple nodes. Based on 9 nodes, that means about 12811/node/s for distributed transactions combined with 115303/node/s for single node transactions.
These were not apples-apples benchmarks, but Clustrix performed about 12,000 multi-node transactions per node per second, along with a whole bunch more single-node transactions.
I don't know how it would perform on the benchmark I used. And I intend to do a tpcc benchmark once InfiniSQL is capable of complex keys (among whatever else it it currently is missing).
Well, array=battery backed controller with a bunch of disks hanging off of it. Actually, there is latency associated with every sync. What I've seen on HDS arrays with Linux boxes and 4GB fibre channell adapters is about 200us per 4KB block write. That is very very good for disk. It's also slower than memory access by many orders of magnitude. This was about 3 years ago. Things are bound to be faster by now, but still not as fast as RAM.
I don't think it's unreasonable to want to write faster than an I/O subsystem can handle. Maybe it's not for every workload, but that doesn't mean it's for no workload.
The distinction I wasn't being clear about was that the storage array (the thing with the RAID controller, cache and disks hanging off of it) is not being saturated if a single transaction log file is being continuously appended to. But that headroom in the array does not translate to more throughput for the transaction log. I don't know if it's an important distinction.
Hi, eksmith. I talk a bit about plans for durability in that overview document.
I promise that I have every intention of making InfiniSQL a platform that does not lose data. I have a long career working in environments that demand 100% data integrity. If I de-emphasized it, it was not intentional.
PostgreSQL doesn't scale for OLTP workloads past a single node. There are a handful of products similar to InfiniSQL (google for the term NewSQL for a survey of them).
And yes, a redundant UPS-backed in-memory cache. I have some ideas on how to do regular disk backing as well (which I'm sure you've read).
And if a more traditional log-based storage layer is added, InfiniSQL will still scale nearly linearly across nodes horizontally. Multi-node scale and in-memory are not dependent on one another. Though I believe that redundant UPS systems managed by a quorum of administrative agents, and provide durability just like writing to disk.
Are you familiar with high end storage arrays, such as from HDS or EMC? They write to redundant memory, battery backed and managed by logic in the arrays. I'm just moving that type of design to protect the database application itself, up from the block layer.
And some people trust their datacenter power--they use pure in-memory databases without UPS already, or they do things like asynchronously write transaction log, which also sacrifices durability. For those groups, InfiniSQL ought to be just fine, without UPS systems.
I totally agree--the architecture I'm calling for is to have redundant UPS's, each managed by InfiniSQL processes--for ultimate availability. If people just want high performance but want to live with a datacenter / cloud provider's ability to maintain power, then I want to support them in that, too.
But I've suffered power outages in data centers, and they'll eventually come around to bite everybody.
There's 3 main reasons for this:
1) there is a lot of conceptual and algorithm stuff that is drummed into people at school that, while possible to learn on one's own, requires time and effort. If I had time to study Knuth in depth, I would love to do so. This subject matter is very often useful--in hindsight, this is my biggest regret for not having studied CS, because there are gaps in my knowledge that would have been easy to fill had I been studying these things long ago.
2) while employers generally do not require a CS degree, it is very often that their employment screening steps will filter you out if you cannot regurgitate much of the stuff described in step 1: algorithms and coding styles taught in college, the way that colleges teach them. No matter how much PR a company claims for itself in looking for intelligence, creativity, and experience, if you haven't internalized the textbook stuff, then you'll frequently get filtered out of interview process quite early. You'll still be able to get jobs, but just not likely the ones with the biggest marquee names with the highest candidate:opening ratio.
3. If you want to do a startup, you'll want to be in the vicinity of people who also want to do a startup. That's much easier when you're in college, and you get looked upon favorably if you come from a prestigious institution. Seed and angel investors like throwing cash at pairs of kids from good schools who can make something that seems useful and is in a hot space.
So, yes, go to college. Study CS, math and also some business stuff like econ. Avoid humanities like plague.