Urllib3 in 2022(sethmlarson.dev)
sethmlarson.dev
Urllib3 in 2022
https://sethmlarson.dev/urllib3-in-2022
115 comments
I'm at Google and I wish each engineer got some kind of budget (like the peer bonus budget) that could be used to distribute funds to third-party libraries. My project is itself open source, does not run in the Google prod environment, and thus is heavily reliant on other open source libraries. I wish I had, say, $1,000/year to personally distribute amongst our dependencies. I think what's holding this back is how logistically/legally complicated it is, especially considering how many of the donations are going to be international and might have tax consequences requiring issuance of a 1099. It's easy to donate to non-profits through work (we have an annual amount we can give, plus matching funds), but for donations to individual developers, the logistics are nigh impossible.
Given that you are a Google engineer, you could probably just pay the $1000/year yourself? That sounds weird at first, because why should you pay for Google's costs. But then again the reasoning is that Google should pay, because they depend on these libraries. But you depend on Google, Google depends on the libraries, so you depend on the libraries. So you not paying is pretty much the same thing as Google not paying: I can understand very well why you don't; because you don't have to.
Some problems with that:
Why should I personally pay for the tools I use at work to make my employer money? Should I also buy my own work laptop? My own software licenses? It doesn't make sense.
Then there's also the tax issue, as others have pointed out. I'd be giving post-tax money in a lot of instances, whereas for Google it would have much more favorable tax treatment.
But the biggest problem is that me personally making $1000/year of donations is pissing in the ocean. It accomplishes nothing. A program where every engineer is doing so accomplishes a lot more.
Why should I personally pay for the tools I use at work to make my employer money? Should I also buy my own work laptop? My own software licenses? It doesn't make sense.
Then there's also the tax issue, as others have pointed out. I'd be giving post-tax money in a lot of instances, whereas for Google it would have much more favorable tax treatment.
But the biggest problem is that me personally making $1000/year of donations is pissing in the ocean. It accomplishes nothing. A program where every engineer is doing so accomplishes a lot more.
> Why should I personally pay for the tools I use at work to make my employer money? Should I also buy my own work laptop? My own software licenses? It doesn't make sense.
Wait til you hear about the amount of tools trades people have to buy for their jobs!
Wait til you hear about the amount of tools trades people have to buy for their jobs!
Yeah I know about that, and it's terrible and shouldn't happen. Thankfully that plague doesn't affect our industry, and I'd hate for it to start.
(Also worth pointing out that some of those tradesmen are legitimate contractors, and own their own tools in the same way that a legitimate software engineer contractor would own their own computer and IntelliJ license.)
(Also worth pointing out that some of those tradesmen are legitimate contractors, and own their own tools in the same way that a legitimate software engineer contractor would own their own computer and IntelliJ license.)
Only if they are self employed though surely?
It's common for car mechanics and construction workers, especially in specializations like electrical or HVAC, to own and use their own tools. Obviously, better employers provide stipends or some form of compensation.
[deleted]
OP doesn't depend on Google. It is a mutually beneficial arrangement, not one of dependency. Perhaps if Google was the only employer that would hire OP, but odds are they could quickly get a different tech job at a thousand different companies.
Your argument could be used to say that OP should pay out of pocket for server hardware to run their programs that Google uses, which doesn't make sense.
Your argument could be used to say that OP should pay out of pocket for server hardware to run their programs that Google uses, which doesn't make sense.
No matter where OP works, they would depend on open-source software. I think the question of dependency is without a doubt here.
My argument cannot be used for server hardware, because, after all, Google has to pay for that.
My argument cannot be used for server hardware, because, after all, Google has to pay for that.
But they(probably) would depend on different open source software
Because the benefit is for Google. Alternative would be to develop in-house or buy a 3rd party library for higher cost to the company.
I assume because donations would, ideally, be tax deductible through the company (or at least would have no personal tax repercussions) whereas making a donation of a gross cost of $1000 to an employee might mean only donating $500-600 net.
I am sure Google would be happy to deduct this from your salary and donate it instead.
You are making some weird arguments in this thread. I don't understand how you think it's reasonable for employees to personally shoulder the burden of everything they use at work rather than work simply paying for it, which makes a lot more sense and is the way that things are typically done. Start deducting the cost of work tools from employee's salaries and suddenly your employees are going to become very inefficient at their jobs, as they won't want to spend their own money on anything.
Sure, from a strictly logical viewpoint you are correct. There is too much red tape and its really Google's problem to fix. And you giving $1000 wouldn't even solve the larger problem of Google not having a system for you to direct them to make the donations. But in both of those cases the project still doesn't have the $1000. You could always just give it to them, on average you'd hardly notice and you might even feel good about what you've done.
My overall political philosophy here is that individual elective donations solve few problems, and that large-scale collective action (whether by governments or large employers) is a much better solution overall. E.g. SNAP is a lot more effective and efficient at ensuring that people don't go hungry than relying on individual people to continuously choose to donate food/money.
Same for this. One individual elective donation to an open source project doesn't do much. A larger system though can do a lot.
Same for this. One individual elective donation to an open source project doesn't do much. A larger system though can do a lot.
Yes, my argument sounds weird. About as weird as Google paying for software they don't have to pay for.
Maybe rephrasing my argument makes it more palatable to you: Shouldn't we, the guild of software engineers, take care of our own and pay our sub guild of open source developers? Why would we leave that to faceless companies?
Maybe rephrasing my argument makes it more palatable to you: Shouldn't we, the guild of software engineers, take care of our own and pay our sub guild of open source developers? Why would we leave that to faceless companies?
That sounds like just a difference of semantics and accounting of the same notion. In net, your suggestion also results in $1k moving from google to the open source. Although that is modulo tax implication differences, since open source projects have not often been able to secure 501c non-profit status from the US IRS, which would facilitate those sorts of pass-through donations easier.
> In net, your suggestion also results in $1k moving from google to the open source.
It also amounts to a $1k pay cut for employees, which no one will be happy with. We already capture a small amount of our value add to the company in terms of wages; this would just make that a little worse. And no, there is no fixed amount that goes to salary plus open source contributions, necessitating the same overall net sum.
It also amounts to a $1k pay cut for employees, which no one will be happy with. We already capture a small amount of our value add to the company in terms of wages; this would just make that a little worse. And no, there is no fixed amount that goes to salary plus open source contributions, necessitating the same overall net sum.
As does yours. But you are using an economic trick to hide it from yourself. See the "Save More Tomorrow" plan by Nobel Laureate Thaler and Sunstein https://en.wikipedia.org/wiki/Nudge_(book)#cite_ref-21
It is not really different from whether the company pays for your office space, laptop, and health care, or makes you pay for those out of pocket.
It is not really different from whether the company pays for your office space, laptop, and health care, or makes you pay for those out of pocket.
It's not an economic trick. If my employer weren't paying for offices they wouldn't be plowing the rest of that on top of my salary. Indeed it's the opposite; fully remote workers make less.
It would probably be way more tax-efficient for the company to donate,
> I'm at Google and I wish each engineer got some kind of budget (like the peer bonus budget) that could be used to distribute funds to third-party libraries.
One of Andrew Yang's policies when he was running for president was to give every American $100 to allocate to whatever non-profits they wanted. This is really the kind of policy we need to drive open source. I'd love to open source more software myself, but right now the thought have having 25 million people using my code commercially and making maybe 5k a year if I'm lucky isn't super appealing.
One of Andrew Yang's policies when he was running for president was to give every American $100 to allocate to whatever non-profits they wanted. This is really the kind of policy we need to drive open source. I'd love to open source more software myself, but right now the thought have having 25 million people using my code commercially and making maybe 5k a year if I'm lucky isn't super appealing.
It sounds good, but will run into problems when implemented at scale because of greedy people, jerks, and scammers.
You will see a cottage industry of OSS projects that serve only serve the purpose of parting people from their money. This is already something we see in the non profit space. But there the extent of the fraud and the damage it causes is limited because there isn’t enough money to attract scammers en masse.
You will see a cottage industry of OSS projects that serve only serve the purpose of parting people from their money. This is already something we see in the non profit space. But there the extent of the fraud and the damage it causes is limited because there isn’t enough money to attract scammers en masse.
In Seattle we have Democracy Vouchers funded by the city that every voter is sent in the mail and can be used to make campaign contributions.
This led one candidate to hire people to solicit/collect these democracy vouchers. He raised over 350k from democracy vouchers.
When the election came around, far fewer people voted for him than contributed democracy vouchers.
And the program administrative and overhead costs are quite high, approaching the level of total funds distributed to candidates.
Just pointing out that handing out money isn't free (although it is one of the true core competencies of the US federal government), and can have unintended unproductive side effects.
This led one candidate to hire people to solicit/collect these democracy vouchers. He raised over 350k from democracy vouchers.
When the election came around, far fewer people voted for him than contributed democracy vouchers.
And the program administrative and overhead costs are quite high, approaching the level of total funds distributed to candidates.
Just pointing out that handing out money isn't free (although it is one of the true core competencies of the US federal government), and can have unintended unproductive side effects.
Just want to point out that this is pretty much WAI: a candidate who sounded promising but wasn't getting funding got airtime, turned out not so great, and failed.
Saying a person should be in the race shouldn't mean that you definitely think they should win it, especially in a FPTP system.
Saying a person should be in the race shouldn't mean that you definitely think they should win it, especially in a FPTP system.
> This led one candidate to hire people to solicit/collect these democracy vouchers.
This sounds like what you'd expect an actually successful/winning candidate to do as well (explore all net positive fundraising avenues). The issue is, this guy wasn't actually a good candidate, and didn't excel at anything else.
This sounds like what you'd expect an actually successful/winning candidate to do as well (explore all net positive fundraising avenues). The issue is, this guy wasn't actually a good candidate, and didn't excel at anything else.
lmao thank god he did not win.
Maybe an internal voting system, where every SWE gets 1 vote, but they submit a couples projects (as a fallback mechanism), so at the end of the yr all donations get bunched up & paid out together. Then if the money can't go to the first project for whatever legal reason it goes down to the next one of their list.
It's still a complicated system so it'd would take serious buy in. Not to mention the drama donations generate internally and publicly. But it's a noble goal.
$1000 per engineer grant might still be easier and less fragile
It's still a complicated system so it'd would take serious buy in. Not to mention the drama donations generate internally and publicly. But it's a noble goal.
$1000 per engineer grant might still be easier and less fragile
This is part of what Spotify did to determine urllib3 as an award recipient: https://engineering.atspotify.com/2022/04/announcing-the-spo...
This is what Microsoft does, basically. Each month, $10,000 is donated to some project. Anyone who has contributed to any open source repo during that month is eligible to vote. There's a good nomination system, too. The write-ups for each candidate are done by employees.
I still like OPs idea of it being more 1-to-1, where the engineer votes their $1000 on a specific project and that's the project that gets it. It's not just a ranked voting mechanism where a top few get it.
The system would only be there to bunch up "votes" so if multiple people choose one project it gets paid out in one bulk payment w/ a fallback to other projects if that payment fails.
The system would only be there to bunch up "votes" so if multiple people choose one project it gets paid out in one bulk payment w/ a fallback to other projects if that payment fails.
Yeah, I'd be prefer that as well. But I understand why it might be harder to accomplish.
$10,000/month for a company the size of Microsoft is so little, though. And only one award per month is way too few as well. I still like my idea of $1,000/yr/engineer of allocations, with monthly batching on the rewards. Also you'd be able to see which projects are already getting substantial support, and then maybe use that information to reward your smaller dependencies that aren't getting as much attention.
Check out the Open Source Peer Bonus Program: https://opensource.googleblog.com/search/label/peer%20bonus
I'm aware of that program, but I think it has some problems. It only happens twice a year, there's a lot of process involved that dissuades people from bothering, the amount of money involved is tiny, and the awards only go to individual people, not general funds for open source projects. What I'm suggesting would be a LOT more total money and would be a continuous process (make the limit $250/quarter).
Part of the joke here is that Google is borderline allergic to using open source software that it doesn't maintain.
Not remotely true in my experience. Have you worked at Google, and seen the sheer size of third_party?
Yes I boomeranged for a total of around 4 years, (ldap mgraczyk). I am aware there is a lot of third party code, but compared to the rest of the world, google is very averse to open source that it doesn't control. See for example frontend frameworks (react vs lit), ML frameworks (tensorflow vs pytorch), javascript transpilers (closure vs everything else), C++ logging, C++ unit testing, C++ anything basically.
It seems like Google is slightly better about this in the Java world, but for the languages I mostly use Google is more guilty of "Not invented here" than other places I have worked. It's possible most of the code in third_party is written by Google, btw.
It seems like Google is slightly better about this in the Java world, but for the languages I mostly use Google is more guilty of "Not invented here" than other places I have worked. It's possible most of the code in third_party is written by Google, btw.
I don't know what to say other than that my experience here is precisely the opposite of yours. Different areas of the company? I use a LOT of third party libraries in the course of my employment, and third_party in my experience is full of legitimate third party libraries, not ones that are developed by Google.
Anyway, here's the list of dependencies for my project. There's a lot of third-party stuff in it: https://github.com/google/nomulus/blob/master/dependencies.g...
Anyway, here's the list of dependencies for my project. There's a lot of third-party stuff in it: https://github.com/google/nomulus/blob/master/dependencies.g...
I think they're leaning in the direction of "Google doesn't like open source because they made Bigtable instead of using MySQL", which, at least when I was there, was a debate that spawned many centithreads.
Generally, I miss 99% of the internal stuff at Google. Prometheus/Grafana is very disappointing compared to Monarch/Panopticon, for example. I also like Go a lot more than Java. So generally, I had no complaints about NIH when I was at Google. Whether it's the best use of their money is up for debate, but I never found them hostile towards open source. (I was also a third_party reviewer. Lots of people using open-source libraries at Google.)
Generally, I miss 99% of the internal stuff at Google. Prometheus/Grafana is very disappointing compared to Monarch/Panopticon, for example. I also like Go a lot more than Java. So generally, I had no complaints about NIH when I was at Google. Whether it's the best use of their money is up for debate, but I never found them hostile towards open source. (I was also a third_party reviewer. Lots of people using open-source libraries at Google.)
Yeah, that could be it. But in a lot of cases Google is writing its own stuff for real reasons.
Maybe I'm just older / have been in more diverse parts of industry, but I've literally worked at employers where open source software was verboten (this used to be quite common even). Compared to that, my use of dozens of open source libraries on my current project at Google is open source heaven. Sure, a smaller startup might be using even more open source libraries because they don't have the resources of Google to write stuff like Monarch/Panopticon for internal use, but to my eye that's orthogonal on the issue of open source friendliness.
Maybe I'm just older / have been in more diverse parts of industry, but I've literally worked at employers where open source software was verboten (this used to be quite common even). Compared to that, my use of dozens of open source libraries on my current project at Google is open source heaven. Sure, a smaller startup might be using even more open source libraries because they don't have the resources of Google to write stuff like Monarch/Panopticon for internal use, but to my eye that's orthogonal on the issue of open source friendliness.
FWIW $20k is approximately the total monthly cost of an average engineer at Spotify. I am appaled at how those fundamental libraries do not get the >= 8 digit support that they deserve. Surely some MBA people can turn this around. FFmpeg burns more in CPU cost every second across major platforms than all those donations amount to.
> Surely some MBA people can turn this around.
Surely some MBA people are the ones driving currently, so you're probably not wrong, but you and I might share different outlooks on anything turning around. I believe we're expected to be grateful that there was any donation at all.
Surely some MBA people are the ones driving currently, so you're probably not wrong, but you and I might share different outlooks on anything turning around. I believe we're expected to be grateful that there was any donation at all.
I think it's less to do with a small number of companies making big donations like this, it'd be much nicer to have many organizations making even small contributions based on usage/complexity of the project.
This is what Tidelift is trying to do, the biggest barrier to more projects getting the amount of funding that we do from them is the number of businesses wanting that sort of relationship with their open source dependencies. The amount we receive from Tidelift is not typical for most projects since we're a medium complexity project that is a part of almost every Python project dependency list.
This is what Tidelift is trying to do, the biggest barrier to more projects getting the amount of funding that we do from them is the number of businesses wanting that sort of relationship with their open source dependencies. The amount we receive from Tidelift is not typical for most projects since we're a medium complexity project that is a part of almost every Python project dependency list.
That's still $20k above what they have to pay for the right of usage of the library. What donation amount above would you consider reasonable? Or how would you budget Spotify's donation accross all open source projects that they use?
You can dual-license to avoid comercial exploitation. Why then set an expectation on how much should be donated for complete open source projects?
You can dual-license to avoid comercial exploitation. Why then set an expectation on how much should be donated for complete open source projects?
Well, I think it depends on the project itself. If they are asking for money, then they probably need money!
See: https://liberapay.org (and also patreon.com has a number of projects)
The idea is that people that can give would give, and those that can't won't. And who can give can also choose how much they think the project is valuable and how much maintenance it needs. Hopefully that would be covering the needs of the developers at a fair salary. It would be nice if open source projects were more transparent about this as well: "I need X/month to reach a fair salary and good maintenance, and could expand, hire devs and add functionality for Y/month." (milestones in patreon partially fulfill that role).
The problem is when there are others that rely on the work as well. How much should each give, such that the system would be fair? From a theoretical perspective, I think the money should be always coming from the source with the lowest marginal counterfactual return. That is, the organizations who have funds going to investors and other orgs with the lowest ROI should instead divert those funds to [something else], where [something else] is OSS in this case, until an equilibrium is reached and everywhere is operating at equal and optimal ROI. In practice things are not so easy. And this analysis (and financial analysis) tends to only value economic ROI, where we should be thinking of social ROI as well as environmental ROI.
But to give a rule of thumb, for now, I think it would be reasonable to pay
(a) if it is less than a % of the (internal) project cost: a % of the budget of the OSS project, depending on how valuable it is internally (say, 50% if their budget is low, or just 1% if their budget is high);
(b) a % of the (internal) project cost otherwise (e.g. at most 5% of project cost).
Ideally there would be some kind of system or framework to streamline this sort of evaluation and allocation to the devs.
I actually think a whole complementary economic system (i.e. enhancing capitalism or socialism) should be developed around this idea. We should be paying whomever is providing value to society, according to their needs to provide this value (and also give them a good life of course! -- and provide a reward/incentive to do valuable things, even ones that don't currently have an incentive). So some kind of organizations, that could be tied to companies or governments too, would be responsible for evaluating on a reasonably objective basis which projects need money and then allocating it (sort of as an outsourcing of the resource allocation job to specialized entities). But meanwhile individual and voluntary giving is basically that without outsourcing. I try to do this personally through Effective Altruism (which is essentially just that: give effectively) and giving what I can to Open Source and other impactful causes.
See: https://liberapay.org (and also patreon.com has a number of projects)
The idea is that people that can give would give, and those that can't won't. And who can give can also choose how much they think the project is valuable and how much maintenance it needs. Hopefully that would be covering the needs of the developers at a fair salary. It would be nice if open source projects were more transparent about this as well: "I need X/month to reach a fair salary and good maintenance, and could expand, hire devs and add functionality for Y/month." (milestones in patreon partially fulfill that role).
The problem is when there are others that rely on the work as well. How much should each give, such that the system would be fair? From a theoretical perspective, I think the money should be always coming from the source with the lowest marginal counterfactual return. That is, the organizations who have funds going to investors and other orgs with the lowest ROI should instead divert those funds to [something else], where [something else] is OSS in this case, until an equilibrium is reached and everywhere is operating at equal and optimal ROI. In practice things are not so easy. And this analysis (and financial analysis) tends to only value economic ROI, where we should be thinking of social ROI as well as environmental ROI.
But to give a rule of thumb, for now, I think it would be reasonable to pay
(a) if it is less than a % of the (internal) project cost: a % of the budget of the OSS project, depending on how valuable it is internally (say, 50% if their budget is low, or just 1% if their budget is high);
(b) a % of the (internal) project cost otherwise (e.g. at most 5% of project cost).
Ideally there would be some kind of system or framework to streamline this sort of evaluation and allocation to the devs.
I actually think a whole complementary economic system (i.e. enhancing capitalism or socialism) should be developed around this idea. We should be paying whomever is providing value to society, according to their needs to provide this value (and also give them a good life of course! -- and provide a reward/incentive to do valuable things, even ones that don't currently have an incentive). So some kind of organizations, that could be tied to companies or governments too, would be responsible for evaluating on a reasonably objective basis which projects need money and then allocating it (sort of as an outsourcing of the resource allocation job to specialized entities). But meanwhile individual and voluntary giving is basically that without outsourcing. I try to do this personally through Effective Altruism (which is essentially just that: give effectively) and giving what I can to Open Source and other impactful causes.
Spotify did an episode of their Development podcast that touched on this recently. I thought it was quite interesting.
https://open.spotify.com/episode/0erUH7oqqbW5HDUjcnK6cb?si=q...
https://open.spotify.com/episode/0erUH7oqqbW5HDUjcnK6cb?si=q...
Tidelift exists. I don't know why it's not more widely used.
Because it is and has always been intransparent about earnings structure (e.g. what percentage of money of a Tidelift subscription is paid out to maintainers), and thus has never achieved critical mass among maintainers.
It also seems like they have mostly pivoted away towards software supply-chain analytics.
It also seems like they have mostly pivoted away towards software supply-chain analytics.
Spotify doesn't pay silicon valley wages in Sweden.
Oh well, if you want money you shouldn't opt for open source in the first place...
Yes of course, but there's value in this approach for everyone. Look at Linux for example, there's a real benefit for various orgs in creating and funding a foundation to further the kernel project. It's not that I think these libraries should get something like a fair market value for their product, just a rather small (for a multibillion corp) donation plan. It's worth it for the PR / virtue signalling alone.
Donations can be very expensive to a large company.
The labor cost is often huge. If you don't prevent fraud at the $500 level you'll have two problems: Lots of repeated fraud at the $500 level, and people trying to get away with fraud at the $5000, $50000, all the way up to FTX levels of fraud. So you need multiple people in different depts to agree to fund something while other people watch over to make sure the people being funded are not the spouses or shell companies of the approvers. Meanwhile there's paperwork costs to make sure the donation paperwork gets to accounting so accounting gets their tax deduction.
At a "really big company" the labor cost of paying an invoice might be $2000. Now does the average programmer get more than $2K of annual value out of urllib3, given the alternatives (which could involve up to architecting around needing it?)
Before you laugh and say "no way" to the $2K figure, imagine spending half of an hour long team meeting of ten software devs getting paid $100/hr but overhead etc costs the company $150/hr for them to attend that meeting, that's $750 of labor costs just to have the dev team possibly agree to support urllib3. Then its got to go up the chain to someone with signing approval, someone has to chase the paperwork to make sure it happens, frankly $2K is on the very low side of the cost of getting an invoice paid.
Commercial non-free software is "put up with" where its not that some rando piece of required software costs $50 thus the cost of the company including labor and overhead is $2050 and they're happy to pay it, its more similar to if you don't dot all the Is and cross all the Ts that software will cost $10M in some crazy lawsuit, and $2050 is cheaper than $10M, so they pay up or they'll get a huge lawsuit or more likely someone will get fired after the audit, and I'd rather pay $2050 for something worth $50 if it prevents me from getting fired and better yet its not my $2050. But nobody never got fired for not paying for urllib3, so nobody's paying urllib3.
Companies will pay for support and I could see a large company paying $10K/yr for a written guarantee if a dev at that company runs into a use problem or bug with urllib3 then an actual dev from the project will help them up to twenty hours per year or similar, even if no one asks for help. In contrast feel good donations are very expensive so they don't happen often, so trying to run a business off feel good donations doesn't work too well..
The labor cost is often huge. If you don't prevent fraud at the $500 level you'll have two problems: Lots of repeated fraud at the $500 level, and people trying to get away with fraud at the $5000, $50000, all the way up to FTX levels of fraud. So you need multiple people in different depts to agree to fund something while other people watch over to make sure the people being funded are not the spouses or shell companies of the approvers. Meanwhile there's paperwork costs to make sure the donation paperwork gets to accounting so accounting gets their tax deduction.
At a "really big company" the labor cost of paying an invoice might be $2000. Now does the average programmer get more than $2K of annual value out of urllib3, given the alternatives (which could involve up to architecting around needing it?)
Before you laugh and say "no way" to the $2K figure, imagine spending half of an hour long team meeting of ten software devs getting paid $100/hr but overhead etc costs the company $150/hr for them to attend that meeting, that's $750 of labor costs just to have the dev team possibly agree to support urllib3. Then its got to go up the chain to someone with signing approval, someone has to chase the paperwork to make sure it happens, frankly $2K is on the very low side of the cost of getting an invoice paid.
Commercial non-free software is "put up with" where its not that some rando piece of required software costs $50 thus the cost of the company including labor and overhead is $2050 and they're happy to pay it, its more similar to if you don't dot all the Is and cross all the Ts that software will cost $10M in some crazy lawsuit, and $2050 is cheaper than $10M, so they pay up or they'll get a huge lawsuit or more likely someone will get fired after the audit, and I'd rather pay $2050 for something worth $50 if it prevents me from getting fired and better yet its not my $2050. But nobody never got fired for not paying for urllib3, so nobody's paying urllib3.
Companies will pay for support and I could see a large company paying $10K/yr for a written guarantee if a dev at that company runs into a use problem or bug with urllib3 then an actual dev from the project will help them up to twenty hours per year or similar, even if no one asks for help. In contrast feel good donations are very expensive so they don't happen often, so trying to run a business off feel good donations doesn't work too well..
> The labor cost is often huge.
In a big company it's often a hassle to pay something to a "new vendor". It usually takes a long time.
> If you don't prevent fraud at the $500 level you'll have two problems
I had a colleague who used to work in the Auditing section. He had various stories of people committing fraud (stories of 50k USD+). There's a cost to trying to prevent all cases of fraud and that might hurt more than to try and prevent everything.
I've also heard cases (not from the auditing guy) where a vendor notified that we paid them 1000 times too much (caused by an input error with the , and the .). That happened despite several layers of checks.
> At a "really big company" the labor cost of paying an invoice might be $2000.
At a "really big company" if something is a good idea then it'll get done. And those handling administrative tasks are often "outsourced" to cheaper countries, e.g. India/Philippines/China/etc.
There have been several donations to "good causes". Those were handled without trying to figure out the cost of someone working in an entirely different part of the world and company. This as it's not like someone will specifically will be hired, or it really matters.
They usually made an event out of an donation (e.g. handing out something in person in an office). Funny enough the event surrounding that was often poorly attended.
> that's $750 of labor costs just to have the dev team possibly agree to support urllib3
You're not talking about a donation any more.
In a big company it's often a hassle to pay something to a "new vendor". It usually takes a long time.
> If you don't prevent fraud at the $500 level you'll have two problems
I had a colleague who used to work in the Auditing section. He had various stories of people committing fraud (stories of 50k USD+). There's a cost to trying to prevent all cases of fraud and that might hurt more than to try and prevent everything.
I've also heard cases (not from the auditing guy) where a vendor notified that we paid them 1000 times too much (caused by an input error with the , and the .). That happened despite several layers of checks.
> At a "really big company" the labor cost of paying an invoice might be $2000.
At a "really big company" if something is a good idea then it'll get done. And those handling administrative tasks are often "outsourced" to cheaper countries, e.g. India/Philippines/China/etc.
There have been several donations to "good causes". Those were handled without trying to figure out the cost of someone working in an entirely different part of the world and company. This as it's not like someone will specifically will be hired, or it really matters.
They usually made an event out of an donation (e.g. handing out something in person in an office). Funny enough the event surrounding that was often poorly attended.
> that's $750 of labor costs just to have the dev team possibly agree to support urllib3
You're not talking about a donation any more.
Could this be outsourced?
I could create a tool - or a suite of tools - to parse e.g. a Python, PHP, Javascript (Node), Java, or other codebase and determine the dependencies. Hell, the requirements.txt file would probably be enough. Maybe I'll manually see how they use Linux, Firefox, LibreOffice, Anki, Inkscape, etc too. The company then cuts me - a reputable business that distributes funds fairly to open source projects - a check and I divvy it up, keep n% for myself, and distribute it to the projects and to the FSF.
With enough clients some open source projects could see significant funds increases, and the companies would be alleviated of responsibility and worry. I could even issue "badges" to significant donors.
I could create a tool - or a suite of tools - to parse e.g. a Python, PHP, Javascript (Node), Java, or other codebase and determine the dependencies. Hell, the requirements.txt file would probably be enough. Maybe I'll manually see how they use Linux, Firefox, LibreOffice, Anki, Inkscape, etc too. The company then cuts me - a reputable business that distributes funds fairly to open source projects - a check and I divvy it up, keep n% for myself, and distribute it to the projects and to the FSF.
With enough clients some open source projects could see significant funds increases, and the companies would be alleviated of responsibility and worry. I could even issue "badges" to significant donors.
> At a "really big company" the labor cost of paying an invoice might be $2000
If that's the case all those credit card bills from corporate trips to other offices must have costed A LOT, like billions. Maybe even hundreds of billions.
If that's the case all those credit card bills from corporate trips to other offices must have costed A LOT, like billions. Maybe even hundreds of billions.
Yeah open source is all about working for free for other people and not about anything else. We can read the story of why RMS came up with the idea of copyleft and it boils down to "I wanted to work for free for other people" and definitely not anything else.
That’s not the point.
Yeah the amount of financial support we receive is mostly dependent on individual organizations. For example in 2021 we didn't receive any large donations, only support from Tidelift.
It would be good to see more large companies making these sorts of donations. It's not like Spotify is unique in its usage of Urllib.
That would be nice, but I think it would be even better if these companies allowed their employees to contribute. And were open about their policies.
Maybe there should be a ranking site where employees can report anonymously about company policies. And companies can respond with their official policies.
The ranking could be something like this
0 - can't contribute at all
1 - can contribute in spare time with approval
2 - can contribute in spare time without approval
3 - can contribute on company time with approval
4- can contribute on company time without approval.
That would be nice, but I think it would be even better if these companies allowed their employees to contribute. And were open about their policies.
Maybe there should be a ranking site where employees can report anonymously about company policies. And companies can respond with their official policies.
The ranking could be something like this
0 - can't contribute at all
1 - can contribute in spare time with approval
2 - can contribute in spare time without approval
3 - can contribute on company time with approval
4- can contribute on company time without approval.
0, 1 and 2 - companies pay me for my time but when they don't pay, it's none of their business what I do, so what's the point?
If you’re an employee (not contractor) in California, your employer can specify in the employment contract that they own IP rights to anything you make even in your own time, as long as it “relate[s] [..] to the employer’s business, or actual or demonstrably anticipated research or development of the employer”. Many tech companies do so, including large tech companies whose business relates to just about everything.
YMMV if you’re in another jurisdiction.
YMMV if you’re in another jurisdiction.
Wow, interesting and horrifying.
A lot of companies have language in their employment contracts that amounts to them owning all of your ideas, even if they are thought of and worked on outside of company time. Under such a contract, you would need to get the company’s blessing to work on some open source project even in your spare time.
Is it not a safe assumption that 1 and 2 are numpy and pandas? I would have thought your question should be phrased “I wonder what the third”.
(Just in case this comes across as snarky, it’s a legitimate question.)
(Just in case this comes across as snarky, it’s a legitimate question.)
I don't know about the third, but requests is probably tied with urllib3 - or does anyone use urllib3 directly these days?
I think boto3 would be at the very top.
If other large companies are considering donating we'd love to help with thanks.dev https://youtu.be/WX3_-cuDXgA
Note that the popular "requests" module depends on urllib3.
This space is a little messy for python. The old urllib ships with python, but it doesn't support some normal expected things like HTTP/1.1 pipelined requests. It adds a "Connection: close" header to each request.
This space is a little messy for python. The old urllib ships with python, but it doesn't support some normal expected things like HTTP/1.1 pipelined requests. It adds a "Connection: close" header to each request.
Relevant to that is the fact that the author of urrlib3 stopped collaborating with the author of "requests" because the latter was misusing the popularity of the library in fundraising for what seemed entirely personal goals, while advertising "new features in requests" that where in fact just urrlib3 features he planned to also expose in "requests". That and a bunch of other stuff, the blog is here:
https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
I think both of them should be in the standard library at this point. The API has been fairly stable for over 5 years.
stdlib is where things go to die. Backwards compatibility, limits on dependencies, release cycles, etc all make development a pain and discourage potential contributions. Or so I've read.
Here's one discussion about it in the requests repo https://github.com/psf/requests/issues/2424
Here's one discussion about it in the requests repo https://github.com/psf/requests/issues/2424
I read that at the time, but that was 7 years ago. Requests has been pretty much backward compatible for that whole time. They could still have active developments available in pypi for people who need the newer features, but python has been dropping minor releases about once a year, that should be plenty fast for new features in requests.
Stdlib may not be the answer, but the current setup does make urllib3 suffer in anonymity...since for many it's hidden under the requests module.
stdlib is also where things go to get used, though.
Coding against a pile of external python modules is perfectly reasonable when you're building something "app scale" -- i.e. something that is going to be spread across many files and installed in a container or venv.
However, when writing something at "script scale" I just don't want to deal with all of that. I want to write something that I can deploy as a single file and not end up dealing with missing python dependencies every time. This means I'm using the old urllib and such more often than I'd like.
It's a shame that there doesn't seem to be much flow of functionality into stdlib at all any more. For my needs, I wish "requests" and "yaml" were in that set, although I'm sure other people have their own opinions on that.
Coding against a pile of external python modules is perfectly reasonable when you're building something "app scale" -- i.e. something that is going to be spread across many files and installed in a container or venv.
However, when writing something at "script scale" I just don't want to deal with all of that. I want to write something that I can deploy as a single file and not end up dealing with missing python dependencies every time. This means I'm using the old urllib and such more often than I'd like.
It's a shame that there doesn't seem to be much flow of functionality into stdlib at all any more. For my needs, I wish "requests" and "yaml" were in that set, although I'm sure other people have their own opinions on that.
Web and release cycles don’t always go hand in hand, though.
Does using requests buy you much on top of using urllib3 directly?
I could look for myself, but it might be more fun as an open discussion topic.
I could look for myself, but it might be more fun as an open discussion topic.
Ergonomics, mostly. To make a basic POST request w/ a JSON body in urllib3, you write:
import json
import urllib3
http = urllib3.PoolManager()
data = {'key': 'value'}
encoded_data = json.dumps(data).encode('utf-8')
r = http.request(
'POST',
'http://httpbin.org/post',
body=encoded_data,
headers={'Content-Type': 'application/json'}
)
import requests
requests.post('http://httpbin.org/post', json={'key': 'value'})In v2.0 urllib3 it's:
import urllib3
urllib3.request("POST", "https://httpbin.org/post", json={"key": "value"})That's fair- I was summarizing latest stable: https://urllib3.readthedocs.io/en/1.26.13/user-guide.html#js...
Does the variable "r" in your first example afford you anything more than what you can get from requests.post? e.g. Can you capture a return value from requests.post, and does it give you the same information to help debug that http.request gives?
> e.g. Can you capture a return value from requests.post
Yes, I just didn't in my toy example. That's the response object, which will typically include all the response info, plus reponse.request, which has request info.
> and does it give you the same information to help debug that http.request gives?
I'm not positive, but I'd bet it's similar.
Yes, I just didn't in my toy example. That's the response object, which will typically include all the response info, plus reponse.request, which has request info.
> and does it give you the same information to help debug that http.request gives?
I'm not positive, but I'd bet it's similar.
Insane that urllib3 only raised $26k, given how it's used EVERYWHERE.
If 1.63% of PyPi is worth $26615/yr, the implication is all of PyPi added together would only be worth $1.6M, which seems low.
Its interesting to compare to other industries. "In the old days" the way to make money during a gold rush was not to try to mine gold after the good stuff was already dug up, but to sell shovels to wanna-be miners. Apparently, software is nearly the opposite of that, LOL.
Its interesting to compare to other industries. "In the old days" the way to make money during a gold rush was not to try to mine gold after the good stuff was already dug up, but to sell shovels to wanna-be miners. Apparently, software is nearly the opposite of that, LOL.
> the way to make money during a gold rush was not to try to mine gold after the good stuff was already dug up, but to sell shovels to wanna-be miners
See also: cloud computing
See also: cloud computing
Thank you for posting this Seth! I will be adding Urllib3 to our (Sourcegraph's) Open Collective budget asap.
Urllib 3 2? I am confused about the version numbers now, wasn't urllib3 the next version of urllib/urllib2?
https://stackoverflow.com/a/63233379/204381
urllib2 was a standard library replacement for urllib in Python 2. Python 3 further replaced this with a new base urllib. urllib3 is a completely separate, unofficial, alternative library.
urllib2 was a standard library replacement for urllib in Python 2. Python 3 further replaced this with a new base urllib. urllib3 is a completely separate, unofficial, alternative library.
The name isn't straightforward, is it? :) The project name "urllib3" is an unfortunate joke on "urllib" and "urllib2" in the standard library, since it's made to supersede both of those modules in functionality.
The original author of the package likely didn't know it'd grow to be the most downloaded package at the time!
The original author of the package likely didn't know it'd grow to be the most downloaded package at the time!
Thank you for the concise explanation. I am an only-occasional Python programmer and did not really understand what was going on with the naming.
Coming soon: urllib 3.2 gen 1 2x2.
Who in their right mind would ever go for such a broken version scheme though. Anything as universal as urllib3 should not need so many details. You should be able to take any latest version of urllib3 and have it all. Anything more complicated will lose many devs for sure.
Universal, that's what the U in urllib is for, right? If you make specific derivatives, it's not universal anymore. Something fundamental is lost.
Full backward compatibility is all we need. Keep it simple please.
Universal, that's what the U in urllib is for, right? If you make specific derivatives, it's not universal anymore. Something fundamental is lost.
Full backward compatibility is all we need. Keep it simple please.
[deleted]
While I appreciate Indeed sponsoring, but why $420?
Are you kidding Indeed?
Your entire business is build around web scraping and using libraries to pull data from other sources. Common, you can do better!
Your entire business is build around web scraping and using libraries to pull data from other sources. Common, you can do better!
Picking on one of the few companies that did contribute something will not help the cause. If more of the companies did likewise there would be real money in FOSS instead of peanuts.
Thank you Indeed (and others on the list, especially Spotify).
Edit: toned down a bit.
Thank you Indeed (and others on the list, especially Spotify).
Edit: toned down a bit.
This is exactly why it's better to not participate. When you participate and people see your name they'll give you grief.
This is the Copenhagen Interpretation of Morality.
This is the Copenhagen Interpretation of Morality.
Some discussion of this phenomenon here: https://news.ycombinator.com/item?id=31720356
Man, How do you remember all this stuff? - "Copenhagen Interpretation of Morality"
Thanks for posting this.
Thanks for posting this.
420 is a weed joke.
In many cases it is difficult to get funding for ad-hoc donations such as this, let alone the authorization to do it on behalf of the company.
I wouldn't be surprised if that effort was driven by a single employee.
I wouldn't be surprised if that effort was driven by a single employee.
Some companies give employees an annual budget to be directed to charitable donations.
I've found that companies that do offer that often restrict it to an established list of charities, which is understandable but means you cannot donate to a random guy in Ukraine maintaining a vital open source project.
Well, thanks to the employee whoever it was
Most likely as a joke, such as $69. Posting publicly as an "inside" joke and to be seen with this sense of humor.
I’ve been writing a library to interact with an api and used the built in urllib.
Is that a mistake? Should I switch to urllib3?
Is that a mistake? Should I switch to urllib3?
If it works and the interfaces are suitable for your needs I wouldn't bother adding an external dependency.
It's something to be aware of should the above change I suppose:)
It's something to be aware of should the above change I suppose:)
It would be good to see more large companies making these sorts of donations. It's not like Spotify is unique in its usage of Urllib.