The title of the book includes "from scratch" for a reason -- it's from "first principles" where you learn about something by building it up from scratch rather than using an implementation. At the end of each chapter, Joel points out the existing resources you can use after learning about the topic.
This has been the most common question I've gotten. I'd say partially it's my bias as someone who works in recommender systems a lot. Many/most common statistical problems have a compact matrix representation (e.g., systems of linear equations). Finally, everyday matrix decomposition tasks like SVD.