Perhaps we just need an S3 implementation specifically built from the ground up for the data-intensive workloads? Such implementation might use conventional S3 underneath for cost and reliability advantages. Such implementation can also deliver a lot of additional functionality which will also be uniform across cloud vendors like enforcing Iceberg conventions on the server-side rather than on the client side and those pre-conditions uniformly implemented or even a full blown transactions but without breaking S3 semantics? Metering and billing can be also made friendly to those data intensive workloads. The question is: if the overhead of such indirection level, right on the data path, can be made reasonable? This is what we are trying to figure out at Embucket.com
I wonder why no one mentions bitonic sort? If you want to do anything in SIMD you better avoid branching as much as possible... and ideally altogether. Here is an implementation I co-authored some 10 years ago: https://github.com/zerovm/zerovm-samples/blob/master/disort/...
Sorting-networks which were already mentioned seems similar but a bit too abstract.
My code above doesn't contains values but those are easy to add I think. Of course it is better to permute fixed size pointers / offsets and not the entire blobs which can be of variable size and then it will complicate everything beyond feasible for SIMD
It is claimed to be also an open-source project and meritocracy is common with open-source projects and github has the required automatic infrastructure for this "keeping track".
I think by "something" he means "phenomenon" and by "re-creating" he means "observing". So I think he is saying: if it is claimed that phenomenon X exists but there is no way to ever observe it for entity Y then X doesn't exists for Y.
But I'm not sure, it's my interpretation. The write-up resonates with me but I can see it as a bit ambiguous and inviting many incompatible interpretations.
Proper treatment of the subject in a book format would be great.
Google is very different from Intel on these fronts at least:
1. Google's open framework TensorFlow is exploding in popularity effectively moving the common interface up from CUDA.
2. Google has a huge need for internal AI/DeepLearning
3. Google has cloud and AI/DeepLearning as a service business
4. Google doesn't sells hardware
However, Intel dominance is not to be underestimated, they definitely can make industry wide impact quickly. Just saying you can not easily draw parallels here
we are heads down coding and haven't got time yet to update ZeroVM website. If you want to contribute contact us now please, if you to use ZeroVM hypervisor stay tuned...
but instance store is transient! You may argue if you do triple replicated in different availability zones then you are ok. Well, in this case it would be very costly as you will end up with constantly spinning EC2 cluster. Even if you don't do any computation you must keep it all spinning. And see what happened to elasticity... you end up paying inflated cloud prices for constantly spinning fixed size EC2 cluster! Instead of being able to rapidly roll out large cluster, make the computation and fold it back and pay only for what you have used - isn't it the true promise of cloud?
Every system might have weaknesses. What matters is:
1. Small surface for attack. With NaCl it is all concentrated in single tiny validator module. The model is also simple and mathematically proven to be secure.
2. Prior testing. It is especially hard for security product. Establishing motivating prizes is good way to ensure it is not easily breakable.
3. The speed with which patch is made available
4. Defense-in-depth, ability to have multiple levels of defense cheaply.
1) there is $100K bounty on each Chrome/NaCl exploit and we have only one ZeroVM 'syscall' that we allow with a lot of attention put how to make it easily secure. The situation is not same on Linux. First of all kernel exploits by process are not really considered severe in Linux and for sure it is not top priority to anyone. Linux built to be secure from outside not from inside.
2) All these is impossible in ZeroVM except accessing memory randomly and thrashing caches and TLB tables. Hm... that could work, I guess. For the first time in this forum we talk about real vulnerability. However, I think the problem exists also in KVM/XEN (will do a proper research now, Googling EC2 TLB thrashing doesn't yield anything interesting), no access to other tenant data just temporarily slowing down specific processor chip.
Well... this echoes ZeroVM ideas but so is PiCloud and a few others mentioned here. I think it goes without doubt that current OSes and VMs are not best suited for cloud technologies. How they can be? The were designed to completely different requirements.
I assume by VM you mean ZeroVM. Well, ZeroVM currently doesn't allow self modifying code at all. Nice try.... We haven't touched Google's provided validator in order not to break anything security-related. And if you think you have a good idea for vulnerability then you can claim some Google prizes.
If you meant more practical uses for it then unfortunately modern JIT would be difficult to support efficiently as they constantly recompile and with ZeroVM it is not only recompilation but also validation. However, JIT that recompile only once, on loading, is easy to support. In fact, next version of NaCl dumps GNU toolchain in favor for JITy LLVM, but then recompilation is happening only once.
you got (2) wrong, the reason is running code on S3 is insecure and would not be allowed and spawning VM inside S3 for some local calculation would be cumbersome at best as it is too bulky for such acrobatics.
If you use EMR or just roll your own Hadoop in EC2 then:
1. Hadoop runs on EC2
2. Data is stored on S3
3. Intermediary results stored in EC2
4. Hadoop loads the data from S3 to EC2
5. EC2<->S3 bandwidth is not that fast or efficient (S3 proxy, network contention, TCP/IP processing)
Hypothetical MapReduce/ZeroVM/Swift scenario:
1. Data is stored on S3/Swift
2. Map and Reduce functions are run inside S3/Swift secured by ZeroVM in majority of cases accessing data locally without networking/proxies getting in the way.
3. Intermediate and final results are also stored within S3/Swift.
4. Local data access is efficient, fast and predictable
5. Local networking within S3/Swift is more efficient, fast and predictable than S3<->EC2 / Swift<->Nova
Accelerated Hadoop scenario:
Exactly as in #1, just Hadoop makes "predicate pushdown optimization" into S3/Swift secured by ZeroVM.
Regarding 'due to security restrictions' I meant that cloud vendor would not let you run your own code in S3 or CloudFiles. Why? Because you could mess up other people data and storage system itself. Why not run in VM inside S3? well I guess it would be impractical due to long provisioning time of conventional VM.
What if I DoS attack some syscall? Or create zillions of files with 1 byte size driving crazy file-system or anything else.
Kernel is such vast area vulnerable for an attack that it is scary even to think about securing all of it and not leaving a single weak point. Moreover, you will screw your syscall API to the point that it will become unusable. At bare least we need standard for the syscall capping and etc... so programmer will know what to expect.
And thanks for the link, will check them and what solution they use and whether they are happy with it.
The answer is that multi-tenant is different from multi-user on so many aspects.
Multi-user:
1. Usually closed institution or even single group within single institution where all know each other.
2. Usually a lot of shared data and application.
3. The risk of malicious activity is low.
4. The consequences of malicious activity are not severe/fatal.
5. Auditing followed by identification and penal actions are an effective strategy to stop malicious activity.
Multi-tenant:
1. Usually open to anyone without "background checks".
2. Single instance of malicious activity is severe, repeated malicious activity is fatal.
3. Auditing, identifying the intruder and suing him in court is not an effective strategy to stop malicious activities.
1. Filtering all resources accesses, letting some pass and others denied.
2. Enforcing different abstraction and then unwanted resources accesses become impossible as they are not even addressable.
Filtering is by definition less secure. As filtering get more complicated there would be false negatives and false positives. Both are harmful.
Enforcing different abstraction is usually less efficient as there is a need to simulate hardware devices. However, some devices have hardware support for virtualization as with Intel CPUs and MR-IOV devices and then enforcing abstraction is free.