I was posting about a similiar system around a year ago.
Ended up standardizing clusters of only 3 nodes. Initially I wrote a Paxos+Dynamo but it was a dead end for me for our needs. I did not need a massive data store, but a cheap highly available / scalable enterprise service that consistently gave the same answer. e.g. an eCommerce pricing engine. 3 commodity box nodes is all you ever need. I've done over 500,000 prices per second with ~ 9 unique way of pricing ~475,000 skus, based off the equivalent of 100's millions of data rows.
Use IP multicast for cluster/leader election configuration. Much faster. All a server needs configured to boot strap into the cluster is a single IP. Opted to not use Zookeeper though it came out when I was writing my system.
I tried a Dynamo/Cassandra approach on the updates but abandoned it as too slow early on. I ended up using something more like Linked-Ins Kafka for the "log" shuttling. In Kafka parlance, one topic with 3 partitions one for each node. Each node mirrors all three partitions but is owner of 1. One other diff is I do push/push while Kafka a does push/pull. This makes sense as each node is both a broker for and consumer of the other nodes.
Node 0 writes incoming updates to partition 0 and does a 1 out of 2 push to partition 0 on nodes 1 and 2 (third auto drops to async as we have 2 of 3 durable stores)
For update application Paxos is used to then only for consensus of the highwater mark of message Id (the kafka offset). Analogous to Spinnaker's LSN.
Consensus updates to the message high water mark are "batch" applied by a single thread, and only a single thread, to the in-memory store which uses a special array based hash map (Compact Hashmap). The single thread with a ReaderWriter lock give transactions for free with a "database lock" being faster row locking as its all in mem. I have the equivalent of 100s millions of rows in mem all synched between the three nodes.
One final optimization is the observation that if the overlaying application is REST based then updates commute under PUT semantics (version with a vector clock). For this type of application the total ordering of the application of the updates across the nodes is relaxed as it is guaranteed to converge (eventually consistent).
Since all update are batch applied, in mem, on a single threads with a reader/writer lock a client always views only the converged consistent state though each nodes path to convergence may be different.
I had looked at using a Dynamo/Cassandra like Spinnaker but after coding it found it too slow. When I went with a log shuttling push/push similar to LinkedIns Kafka performance of updates jumped quite nicely. I note one of the authors works at LinkedIn.
Wrote the thing in Scala which was also sort of new at the time. But turned out to be a good choice.
Over the holidays I toyed around with scratching an itch I've had with purchasing Kindle books from Amazon. It's just too hard to casual browse through the offerings. So I came up with a small chrome web app.
I'm mulling over taking it to the next level. Thoughts, feedback and suggestions. And yea, I know the UI sucks. It would be first on the list.
The Problem I'm Trying To Solve: Can't browse the long tail of Amazon's Kindle book offerings quickly and easily. I don't want to replace Amazon's shopping experience, just offer a complementary alternative.
It's currently running on an EC2 micro at 3 cents an hour so be gentle. Look at it cross eyed it'll probably fall over.
Very much looks like a nicely done but nevertheless small subset of emacs org-mode for the web.
I'll stay with org-mode:
- I'm in emacs all day anyway.
- Much more feature rich: priorites, dates, coloration, etc.
- Full editing (duh) capabilities. Cut&paste to shuffle things around. Key combos for everything. Fingers never leave the keyboard.
Since I maintain the file in git I check it in and out from anywhere on any system. Full version history as well.
It is / can be surprising who on your team will have the most problems on a Git from SVN migration. Sometimes ol' CVS/SVN warhorses have too much to unlearn, sometimes simple devs who never bothered to progress beyond the minimal clicks with SVN/IDE to get code up and down discover the true Qi of source control management as they discover git's capabilities.
One nice thing about git vs. svn is when you reach enough mastery you can make that repo dance to your tune. At some point you and git are one and working together. SVN was always some kind of imposing, opposing force; one that I had to fool into doing my desire.
One clueless contractor could put your SVN repo into hell-and-gone. With git I told the team, "don't worry, go for it; unless you do something really strange, there's nothing you can do that I can't recover from."
Don't lose another four years before blogging about how you finally switched to emacs from vim and bite the bullet now. Emacs is to vim as vim is to textmate IF you are a hard core software developer. Vim is best for sysadmins, simple, quick non-complex editing.
I know, I know... But when you were newly in "love" with textmate I'm sure if I said you need to switch to vim you'd say no way. Skip vim and go right to the best there is, Emacs.
Chromium + Flash + Linux vulnerable as well?
How does one
a) even know what version of flash is embedded in Chromium
b) other than constantly killing the flash process how does one disable flash in Chromium
Not too mention the basics, eye opening approaches to XML and Web, an IDE, debugger, contracts, typed/dynamic language, jit, Android development, ... and on and on.
The talent of the core PLT group is outstanding. Matthias Felleisen for example was awarded an ACM Fellowship in 2006 for contributions to programming languages and development environments. His academic publications are right up there with the best out there, yet he spends as much time focused on the foundational aspects of teaching kids, and students as high brow papers, and the real world demands of programming. And the rest the core are not too shabby either.
IMHO, there are currently only 2 top tier active hotspots where the cool theoretical meets with the practical and usable in programming language theory, the Haskell and PLT ecospheres.
Scala, and Closure would be next.
Don't listen to the wingnuts comedy central wannabe's cracking poor puns here about schemes and rackets (well laugh at the good ones).
In all seriousness, if you are at that point where you've stumbled onto the fact that there is a whole world beyond Java, Cobol, C and C++ and are having fun exploring Smalltalk, SML, OCaml, Haskell, Scala, Closure et al, do not skip, repeat, do not skip exploring PLT Racket. It is as rich, and deep and mind altering as any of them.
The next generation enterprise system
(the next JBoss if you want to go for it)
- JVM based
- distributed multicore 2-3 nodes => 32 - 96 or more cores
- pluggable/dynamic/module substrate OSGI - Felix, Karaf
- Components written in Scala
-- w liberal use of Actors, some STM as well
- Paxos based node coordination and eventual data consistency
- The majority if not all data in memory, check pointed on SSD.
- Cluster of N nodes survives failures and keeps on chugging. 2F + 1 = N
As the instant after the Big Bang the Waveverse is pretty small. Or so it seems ...
Try this. Go to the search bar, if you're currently viewing your inbox it has "in:inbox", remove that and search "with:public". You are now amidst the maelstrom of hyperinflation of the birthing Googleverse.
A bit quick to skip the trial and go straight to the hanging... As the other poster stated give it time. Look past the SMS / IM social butterfly aspect to the big picture.
I think its the best damn thing I've seen come down the Internet pipe in a loooong while. Hands down.
Does anyone have any idea on how "hard" it is to create software? The toll on a person who is "tearing it up", without break, for a year. Oh, and not for their direct benefit?
You are seeking a person in the upper percentile of intelligence, fluent in a specialized skill set and asking them to pull off some solo Death March that should require 5-10 people minimum, and for what, to be the "coder" while the idea man is running around with half-cent business cards with CEO in front of their name.
Assume this person _is_ this smart, maybe even smarter than you, most likely he too has realized its easier to be the idea man than to be the guy who has to do all the work.
Ergo the world is stuffed with idea men and suffers a dearth of doers.
Scala is _not_ some esoteric MIT, CMU language to make a point exercise. Its a pragmatic and practical attempt to "fix" Java so it has all the capabilities of Java with the proper foundational coherence which allows for additional, some yet to be defined or added, capabilities. Java has long ago painted itself into a corner. It has reached its evolutionary dead end.
As someone who lead a team which wrote one of the largest retail websites in the earlier days of Java, and as someone who has spent the last year rewriting legacy business logic in Scala, there is no comparison.
Scala wins not because its the new hipster language, it's distinctly superior across the board. You can write better, clearer, far more robust, far more scalable and far more easier to maintain and modify business logic at higher levels of productivity. That and its ability to work with legacy Java code makes it ideal. This is from direct and in-depth experience.
My concern with gizmo's post is its complete lack of claimed experience. No context of based on my experience where ... so we ... and then did a POC comparison with ... and found .... and concluded ...
In fact it appears very much to be a direct quote of from the preamble from the "Enterprise IT Managers Survival Handbook For Those Without Experience Or Knowledge Of Software Development Fundamentals"
Privately I've played with all of them. I love the shear outrageous power and beauty of Scheme (PLTs impl) and Haskell, the base competency of SML, and I've even used them in the enterprise in isolated in very few situations where the situation demanded it.
However, never once did it cross my mind to even "think" to introduce any of them into the enterprise (well a brief fling with Python for our operations group, it was found to be too hard).
Scala deserves to win. It is decidedly superior, however, lots of deservedly good things, in fact most, never see the light of day in IT.
In the last 2-3 years emacs has just improved and improved. Like most it originally took me 2-3 distinctive efforts to "go emacs", but it pays off. Emacs is just outstanding if you are a code slinger.
If your an admin and edit a couple of config files now and again, maybe VIM, though IMHO emacs is superior for even simple editing as well.
Never could adapt to OCaml's undisciplined syntax. SML's is clean. MLTon is very nice with a couple of limitations, practical upper bound of program size and compilation times.
One of the co-authors, S. Weeks, was (is?) one of the core MLTon compiler developers. If feasible they certainly could have evolved the compiler and the SML language.
Somewhat interesting is F# lost out to Ocaml in a bakeoff.
In a way Jane St. has got themselves in a bit of a corner. I don't see how Ocaml is substantially less moribund then SML.
Here is an interesting thought to mull over. With the upcoming 2.8 Scala compiler _every_ substantive feature of (S/C)ML is available with equal or higher capability.
In other words, strike out use of objects in Scala and one is left with the most advanced (and active) MLs around. Hmm...