Pretty general question, but what has your approach been for coupling ThreeJS + React w/ a Rust/Wasm kernel for mesh generation? E.g. do you have Wasm own the memory and you give ThreeJS views of the memory to upload to GPU?
Sure I'd be happy to check it out, my email's in my profile (or Github/website).
There are some tradeoffs w/ WebAssembly as well (not sharing the same memory as JS/TS is the biggest one) and debugging can be a bit tough as well though now there's a good VSCode plugin for it [0]. Another part of the reason I also moved back to C++ -> Wasm was for the performance improvement from Wasm vs. JS/TS, but the cross compilation to native/web was the main motivator.
Author of the WebGL volume rendering tutorial [0] you mentioned in the readme here, great work!
Working in WebGL/JS is nice since you can deploy it everywhere, but it can be really hard for graphics programming as you've found because there are very few tools for doing real GPU/graphics debugging for WebGL. The only one I know of is [1], and I've had limited success with it.
WebGPU is a great next step, it provides a modern GPU API (so if you want to learn Metal, DX12, Vulkan, they're more familiar), and modern GPU functionality like storage buffers and compute shaders, not to mention lower overhead and better performance. The WebGPU inspector [2] also looks to provide a GPU profiler/debugger for web that aims to be on par with native options. I just tried it out on a small project I have and it looks really useful. Another benefit of WebGPU is that it maps more clearly to Metal/DX12/Vulkan, so you can use native tools to profile it through Chrome [3].
I think it would be worth learning C++ and a native graphics API, you'll get access to the much more powerful graphics debugging & profiling features provided by native tools (PIX, RenderDoc, Nvidia Nsight, Xcode, etc.) and functionality beyond what even WebGPU exposes.
Personally, I have come "full circle": I started with C++ and OpenGL, then DX12/Vulkan/Metal, then started doing more WebGL/WebGPU and JS/TS to "run everywhere", and now I'm back writing C++ but using WebGL/WebGPU and compiling to WebAssembly to still run everywhere (and native for tools).
With WebGPU, you could program in C++ (or Rust) and compile to both native (for access to debuggers and tools), and Wasm (for wide deployment on the web). This is one of the aspects of WebGPU that is most exciting to me. There's a great tutorial on developing WebGPU w/ C++ [4], and a one on using it from JS/TS [5].
Yeah! I've started to look some WebGPU compute applications with other students in my group and I think there could be some cool use cases, like the ones you mention. It sounds a bit odd, but yeah WebGPU on native (by directly using Dawn, or wgpu-rs) is actually pretty compelling as a cross-platform low-level graphics API.
What's really cool is that compute and rendering using WebGPU can get near-native level performance. So a lot of scientific applications (which typically rely on more FLOPs/parallel processing) can be implemented in WebGPU compute without sacrificing much performance. I'm not sure how many simulations would be ported to WebGPU, since they usually end up targeting large scale HPC systems and CUDA, but for visualization applications I think the use case is pretty compelling, especially for portability and ease of distribution. On the compute side, I implemented a data-parallel Marching Cubes example: https://github.com/Twinklebear/webgpu-experiments , and found the performance is on par with my native Vulkan version. You can try it out here: https://www.willusher.io/webgpu-experiments/marching_cubes.h... . There is a pretty high first-run overhead, but try moving the slider around some to see the extraction performance after that. WebGPU for parallel compute, combined with WebASM for serial code (or just easily porting older native libs), will make the browser a lot more capable for compute heavy applications. You could also combine these more capable browser clients with a remote compute server, where the server can do some heavier processing while the client can do medium scale stuff to reduce latency or work on representative subsets of the data.
ISPC is really awesome, but you're right it is much less known than CUDA/OpenCL. Part of that might just be lack of marketing effort and focus (you don't hear much about it compared to e.g. CUDA) and the team working on it is far smaller than that on CUDA. There has been some wider adoption, like Unreal Engine 4 using it now: https://devmesh.intel.com/projects/intel-ispc-in-unreal-engi... which is super cool, so hopefully we'll see more of that.
As far as support from other languages I did write this wrapper for using ISPC from Rust https://github.com/Twinklebear/ispc-rs (but that's just me again), and there has been work on a WebASM+SIMD backend which is really exciting. Intel does also have an ISPC based texture compressor (https://github.com/GameTechDev/ISPCTextureCompressor) which I think does have some popularity.
However, the domain is pretty specialized, and I think the fraction of people who really care about CPU performance and are willing to port or write part of their code in another language is smaller still. It's also possible that a lot of those who would do so have their own hand written intrinsics wrappers already. Migrating to ISPC would reduce a lot of maintenance effort on such projects, but when they already have momentum in the other direction it can be harder to switch. I think that on the CPU ISPC is easier and better than OpenCL for performance and tight integration with the "host" language, since you can directly share pointers and even call back and forth between the "host" and "kernel".
SIMD is used a ton in rendering applications and starting to see more use in games too (through ISPC for example).
I'd add to the list:
- Embree: https://www.embree.org/ Open source high-performance ray tracing kernels for CPUs using SIMD.
- OpenVKL: https://www.openvkl.org/ Similar to Embree (high-performance ray tracing kernels), but for volume traversal and sampling.
- ISPC: https://ispc.github.io/ an open source compiler for a SPMD language which compiles it to efficient SIMD code
- OSPRay: http://www.ospray.org/ A large project using SIMD throughout (via ISPC) for real time ray tracing for scientific visualization and physically based rendering.
- Open Image Denoise: https://openimagedenoise.github.io/ An open-source image denoiser using SIMD (via ISPC) for some image processing and denoising.
- (my own project) ChameleonRT: https://github.com/Twinklebear/ChameleonRT has an Embree + ISPC backend, using Embree for SIMD ray traversal and ISPC for vectorizing the rest of the path tracer (shading, texture sampling).
A cool feature I recently learned about of matplotlib is that it supports LaTeX for text rendering [1]. You can go as far as rendering LaTeX math formatting for titles/labels, or just have the plot fonts match your text and/or figure captions so it fits nicely into your paper.
This is really cool! Sorry if this is kind of a dumb question but I know nothing about web development: can I use this without running node on the server? I'd like to use this for a WebGL distance field renderer I'm working on but host things on my Github page and thus can only do client side stuff. Getting a server is a possibility but for just one tiny project it doesn't seem worth it.
You're right, these images are rendered with path tracing (I do also have a Whitted integrator). However tray_rust is a ray tracer in the general sense where it refers to a family of methods that all involve tracing rays to render an image, eg. Whitted recursive ray tracing, Path tracing, Bidirectional path tracing, Photon mapping, Metropolis light transport, Vertex Connection and Merging, etc. (since I hope to implement some of these methods as well!).
It sounds like our instance types will end up being a bit different. Right now we don't need to send the hit instance back with the differential geometry since it doesn't tell us anything extra but later we'll attach different materials and lights to instances of the same geometry and will then need access to the hit instance so we can compute the material properties.
Sending the instance to the geometry to fill it out in the struct is a good idea, if Rust had default parameter support I'd do this and have the instance's intersect just default to None and pass itself to the geometry's intersect. Then the functions would have the same signature and we could mark the case of a None instance in geometry as unreachable.
Right, the instances go into a scene level acceleration structure and for each triangle mesh we build an object level acceleration structure on the triangles. I was planning to do something like a BVH<T: Geometry> so then I don't need a separate BVH type for instances and triangles, since Instance implements the Geometry trait. I'm not sure if there's a better alternative that would help get around the DifferentialGeometry issue since in this case the instance members of geometry types (Spheres, Triangles) and Instances must have the same signature.
Author here, I'm not quite sure I follow. Would this then have the Sphere's and Instance's intersect implementations differ, one returning PreDifferentialGeometry and one returning DifferentialGeometry? I thought about changing the instance implementations to return different types but later when we want to put both our geometry instances and geometry itself (triangles in a mesh) into bounding volume hierarchies I think we'd run into trouble.