annowiki

2 jaar geledendiscuss

How do you get around 403/401's from WSJ/Reuters/Axios? Because I've tried user agent manipulation and it seems like I'd have to use selenium and headless to deal with them.

annowiki

2 jaar geledendiscuss

> Another area is with software we’ve had to build (instead of buy). When we started out, we strongly preferred buying software over building it because a team of only a few engineers can’t afford the time cost of building everything. That was the right choice at the time even though the “buy” option generally gives you tools that don’t work. In cases where vendors can’t be convinced to fix showstopping bugs that are critical blockers for us, it does make sense to build more of our own tools and maintain in-house expertise in more areas, in contradiction to the standard advice that a company should only choose to “build” in its core competency. Much of that complexity is complexity that we don’t want to take on, but in some product categories, even after fairly extensive research we haven’t found any vendor that seems likely to provide a product that works for us. To be fair to our vendors, the problem they’d need to solve to deliver a working solution to us is much more complex than the problem we need to solve since our vendors are taking on the complexity of solving a problem for every customer, whereas we only need to solve the problem for one customer, ourselves.

This is more and more my philosophy. I've been working on a data science project with headline scraping (I want to do topic modeling on headlines during the course of the election) and kept preferring roll your own solutions to off the shelf ones.

For instance, instead of using flask (as I did in a previous iteration of this project a few years ago) I went with Jinja2 and rolled my own static site generator. For scraping I used scrapy on my last project, on this one I wrote my own queue and scraper class. It works fantastically.

annowiki

2 jaar geledendiscuss

Actually it seems pretty accurate. Novelty-seeking is a well known phenomenon in curious individuals. https://en.wikipedia.org/wiki/Novelty_seeking

Literally getting dopamine rewards for seeing something new is what keeps people glued to tik tok feeds and twitter.

I tend to get bored halfway through a book if it is predictable.

annowiki

3 jaar geledendiscuss

This is not really "sieve-ing" per the article, but what prevents me from running another process that periodically queries the data in a cache? Like just running a celery queue in Python that continually checks the cache for out of date information constantly updating it? Is there a word for this? Is this a common technique?

annowiki

3 jaar geledendiscuss

There's a piece of graffiti from the May 1968 Paris riots that goes "I have something to say but I don't know what." Reminds me of this. I can't find the original French but I think it's something like "J'ai quelque chose a dire mais je ne sais quoi."

Found it: https://www.dicocitations.com/citations/citation-67489.php

They use the pas

annowiki

4 jaar geledendiscuss

I started as a python programmer and was very used to package managers. I believed in them, I championed them. When I switched to C++ for work I was very disheartened that there wasn't a standard.

Conan obviously has promise, I haven't spent much time with it, most of my experience with C++ package managers is with nuget and vcpkg. However, my attitude toward package managers is changing.

I increasingly like _not_ using package managers because it makes me (and my company) way way way less likely to bloat our software with unnecessary third party dependencies.

I wrote this in another thread: I never believed you should write something yourself if you can find a package for it. My boss told me I should write it all myself, I could probably write it to be faster. I encountered a case where I needed to compare version numbers in python. For the heck of it I wrote the simplest, quickest, most naive solution I could come up with and then timed it against the most recommended version comparison package in python. I blew it away by 20x throughput.

I don't believe in package managers anymore. Obviously I'll keep using pip and sqlalchemy in Python, but I'll happily spend the 20-30 minutes it takes adding something like nlohmann-json or md4c to my project over worrying about maintaining a package manager for c++ these days. Precisely because it makes me think twice about adding another dependency.

annowiki

4 jaar geledendiscuss

A Programmer's Introduction to Mathematics https://pimbook.org/

It introduces math from a mathematician's point of view (complete with proofs, etc.) rather than rote memorization and exercises, but it does so from the perspective of a programmer.

annowiki

4 jaar geledendiscuss

You have me thinking about kind of a cool board idea. 150 person twitter boards. Cap it at 150. People in that group can all vote on their own moderation, they can't interact with groups in other boards through quote tweeting or voting, though obviously they can copy paste.

You might get racist boards, but then its easy to get rid of all of them at once.

150 being https://en.wikipedia.org/wiki/Dunbar%27s_number of course.

I have no way to distribute anything. I tried to do my own annotations board on literature but no one joined. I just think it sounds cool to be in a personable board like that.

annowiki

4 jaar geledendiscuss

When I run ls I don't want to see all the configuration files. Just my files. I think that's the point of hidden files.

> One more example. Imagine if you have a project and want to edit an .env file. But as dotfiles are hidden in Linux you don't see this file and cannot open it.

How likely is it that someone is going to want to edit a .env file and not know how to view hidden files?

annowiki

4 jaar geledendiscuss

When I started playing with C/C++ the first thing I complained about was how difficult package management was. Conan is not as simple as cargo or pip. It's often simpler to find a header only library and plop it into your repo, but you still have to modify build configurations to include it.

This is the first time I thought to myself "Maybe that's actually a good thing."

annowiki

4 jaar geledendiscuss

For our purposes, we never have strings that are other than the form XX.XX.XX.XXXX so there was no reason to generalize. Which improves the speed, makes it easier to read, and provides all the more reason not to use a library.

annowiki

4 jaar geledendiscuss

Its usually not so much "a range of wheels optimized for specific roles" and more "one size fits all."

You want a wheel that fits your cart. You have a lathe. Why make do with a wheel someone else made, standardized, and sells to fit a wide range of carts when you can make the perfect wheel for _your_ cart?

Most of the functionality in libraries use standard algorithms anyway. I doubt anyone thinks its a good idea to write your own cryptography or markdown processor, but why do you need a library to left pad a string with zeroes?

if statements and flags take time to process. You only have one way you need to do something. Do you want to take that much more compute just checking flags that will never change so you can get the library function to do what you always want it to do?

annowiki

4 jaar geledendiscuss

Its a numpy style docstring. PyCharm can be configured to auto generate them. I guess I just add the types to be thorough.

Ideally I want to generate api documentation via a tool. A long time ago I wrote a script to parse the AST and generate markdown from the docstrings and function information, but it was mediocre and I haven't wanted to use sphinx because its too heavy and doesn't seem to produce ideal markdown output.

numpy seems to sometimes use type hints in the docstring, other times not: https://numpydoc.readthedocs.io/en/latest/format.html#parame...

annowiki

4 jaar geledendiscuss

I come from a Python background so libraries were always just a pip install away.

In the past year, however, I've been working on a large C++ codebase (couple million lines) and the result has been a considerably greater amount of "roll your own."

This has filtered back into my Python. If it's not in the standard library, I don't install it unless I really need it.

My boss always tells me that you can probably write it faster than a library, something I never used to believe. Until I tried it. I needed to check which of two version strings, of XX.XX.XX.XXXX format, were bigger. I tried the most recommended version number library, then I tried writing my own solution, the simplest solution I could think of:

    def version_compare(v1: str, v2: str) -> int:

        """
        Compares two versions.

        Parameters
        ----------
        v1: str
        v2: str

        Returns
        -------
        int
            1 if v1 is greater, -1 if v2 is greater, 0 if equal.
        """
        for el1, el2 in zip(v1.split('.'), v2.split('.')):
            if el1 != el2:
                return 1 if int(el1) > int(el2) else -1
        return 0

My code was faster by like 20x. Libraries are bloated and you probably only need a small subset of the functionality, so write your own code has become my mantra.

annowiki

4 jaar geledendiscuss

I believe he means "the use of asterisks to mark required fields" was invented for the web. Otherwise, you'd have to be quite young and quite sheltered to not have seen asterisks in places older than the web.

annowiki

4 jaar geledendiscuss

https://en.wikipedia.org/wiki/Free_Speech_Movement

Mario's "Operation of the Machine" speech is pretty good.

Also worth noting, often these groups were not quite as egalitarian as they're thought to be. SDS for instance had quite a bit of sexism in its operations: https://en.wikipedia.org/wiki/Students_for_a_Democratic_Soci...

I have no position on this debate, I just thought you'd want a little context.

annowiki

4 jaar geledendiscuss

I can strongly recommend Fussel's (the author of this essay) book "The Boys' Crusade", which provides a very eye opening account to WWII. It is most likely a longer version of this essay (I haven't read this essay).

I can also recommend his book Poetic Meter & Poetic Form, but for different reasons.

Other books in the same vein:

- The Warriors: Reflections on Men in Battle by J Glenn Gray (Gray was a philosophy PhD and a 2nd Lieutenant in the war)

- War is a Force that Gives Us Meaning by Chris Hedges (Hedges was a war correspondent during the balkans and numerous other conflicts, this is much more interested in the psychological build up to war in common society, and its effects on society)

These books make me very pessimistic about human nature, but Nicholas Wade's Before the Dawn: Recovering the Lost History of Our Ancestors provides a nice antidote: the total number of people killed by warfare in the 20th Century, if we followed similar patterns to our prehistoric ancestors, would have dwarfed what actually happened. We are becoming less warlike, even if its not cured.

annowiki

4 jaar geledendiscuss

Small typos everywhere. Init functions without the dunders, move_ahead in an abstract class then moveahead later.

annowiki

4 jaar geledendiscuss

I just linked your blog post in a separate comment because I was reading it over the weekend. I was looking for articles about writing static site generators and didn't realize most people don't roll their own templating engines. I was disappointed to see not much on templating in your post but it definitely gave me the confidence to keep going. Managed to get aspects I'd been struggling to work for months working yesterday.

annowiki

4 jaar geledendiscuss

This is mindblowing synchronicity. I thought this article was one I had read this weekend for a second but checked the post date and saw it wasn't available yesterday.

I'm working on trying to write a static site generator for my own personal site/blog in C++, been tinkering for a couple months. It started as a passion project in tribute to my dog that passed. GatsbyJS wasn't working (again) which is what my current site is built with, so I just said screw it, I'll write my own. Chose C to begin with and quickly gave up. Decided to switch to C++ because its what I'm supposed to be learning for work.

I named it bluesky, after my dog that passed, Sky Blue. https://github.com/mas-4/bluesky

Templating is a lot harder than I had initially thought. I wanted the templating system to be as bare bones as possible, similar to another static site generator I rather like, sergey[0]. Most of these articles I found ([1], [2]) about writing your own SSG don't go into templating much, they just use an off the shelf library. Inja[3] is available for C++ but, like I said, I want something really bare bones, like if you were designing html now, you'd obviously include html-includes and html-templates. I finally got it working for includes and templates, now I have to add the markdown support, and then I plan on migrating my personal site to using it.

[0]: https://sergey.cool/ [1]: https://www.smashingmagazine.com/2020/09/stack-custom-made-s... [2]: https://blog.hamaluik.ca/posts/build-your-own-static-site-ge... [3]: https://github.com/pantor/inja

edit: a link

Submissions

Ask HN: Tools for programmatically improving a big legacy code base?

Ask HN: Advice for cross-platform dev environments for Python Extensions

Accounting for Computing Scientists

Floriography (Language of Flowers)

Ask HN: How do you run long computations without interruption?

Semantic Satiation

Show HN: A Wikipedia for Annotations

Show HN: Anno.wiki – Annotation Wikis on Literature

comments