you don't transpose it before the matmul, you always have it transposed (i.e., when you print the weights of a linear layer in pytorch, you're actually seeing (A^t)^t and what's stored is A^t.
a polyhedral compiler wouldn't find this either - polyhedral compilation is for finding optimal schedules for loop nests i.e., the order in which independent (wrt dataflow) iterations run. as far as i know you, a transpose can't be expressed in the polyhedral model.
there is not a single org anywhere in the world that uses pure python to do numerics. kids do that during their first linear algebra or ml class. that's it.
>For example replacing IF conditions by np.where etc
i've already addressed this - this is not jit compilation.
since this post might attract people that are mathematica powerusers: how good are mathematicas optimization routines vs commercial solvers like gurobi or cplex? reason i ask is i'm spinning up project that'll require a good bit of MIP, MILP and i have a mathematica license but i'm considering getting a gurobi license. since i've never tried gurobi i can't compare.
given who you are (googling your name) i'm surprised that you would say this. jax does not jit compile python in any sense of the word `Python`. jax is a tracing mechanism for a very particular set of "programs" specified using python; i put programs in quotes because it's not like you could even use it to trace through `if __name__ == "__main__"` since it doesn't know (and doesn't care) anything about python namespaces. it's right there in the first sentence of the description:
>JAX is Autograd and XLA
autograd for tracing and building the tape (wengert list) and xla for the backend (i.e., actual kernels). there is no sense in which jax will ever play a role in something like faster hash tables or more efficient loads/stores or virtual function calls.
in fact it doesn't even jit in the conventional understanding of jit, since there is no machine code that gets generated anew based on code paths (it simply picks different kernels and such that have already been compiled). not that i fault you for this substitution since everyone in ML does this (pytorch claims to jit as well).
1. this is a thread about cpython. jax is as relevant to users of cpython as CUDA or OpenCL or whatever. jax cannot do absolutely anything with e.g. django.
2. for all intents and purposes all numerical code always runs in a lower-level implementation (C++, CUDA, XLA, whatever). so from that perspective, jax is just a convenient way to get from numerical python (i.e., loops and muls and adds) to kernels.
these two things have nothing to do with each other. jax doesn't compile numpy, it reimplemnts the api using `ufunc`. in general, every single numerical kernel is always mapped to kind of compiled code.