smif · HackerTrans

3 jaar geledendiscuss

I think the point here is that it's subverting and redirecting Bentham's own utilitarianism against itself. How does the utilitarian decide which one of those has more utility? That's a rhetorical question and it's sort of immaterial how that question gets answered, because regardless of how they decide, the dialogue is structurally describing how utilitarianism is vulnerable to exploitation of this type.

smif

3 jaar geledendiscuss

It seems like 1.25 is a month and a half out from retirement, maybe it's related to that.

Kubernetes 1.25 (in Danny Glover voice): I'm getting too old for this shit!

smif

3 jaar geledendiscuss

Your definition of loyalty sounds like it's more loyalty to yourself/your own values or an idealized/rose-tinted view of the company from the past rather than loyalty to what the company is at the present time, which is always a moving target of course.

I interpreted their notion of loyalty as like a sort of trust that things will work out in the end, "maybe this is just a rough patch", etc.

Re: neglect, I think there's a more subtle gradient there. Lets say your company decided to implement a dumbass policy X, and you rail against it, but since it's highly popular it's like you're pissing against the wind. So you kind of resign yourself to it's inevitability and make a long bet that it's going to fail. This is in the hopes of a brighter future where it has already failed and been replaced with something better. You might even make decisions during this time that set you up to be better positioned for when it does eventually fail. This is a kind of negative semi-passivity, but it's only negative in the sense that you're betting against something that you think is a bad idea and you're trying to improve your position within that potential better future after the bad thing is gone.

That's not strictly the same thing as sabotage which would require malicious actions on your part to undermine the effort. You don't even need to hope that it fails, you can just make impartial/rational decisions to prepare for it's eventual downfall without being heavily emotionally invested in that same downfall. And on top of that, you could rationalize all of that as setting the company up for future success after it gets through this rough patch, so it's not all about self-interest.

E: There's a famous chess saying that comes to mind, "never interrupt your opponent while they are making a mistake". Not sure about it's applicability here, but it feels like a similar kind of mindset.

smif

3 jaar geledendiscuss

> Re: ORMs, I respectfully disagree. I’ve come across many teams that treat their Python/Rust/Go codebase with ownership and craft, I have not seen the same be said about SQL queries. It’s almost like a 'tragedy of the commons’ problem - columns keep getting added, logic gets patched, more CTEs to abstract things out but in the end adds to the obfuscation.

> ORMs don’t fix everything but it does help constraint the ‘degrees of freedom’ and help keeps logic repeatable and consistent, and generally better than writing your own string-manipulation functions. An idea I had I continued (I wrote the post early last year) was to use static analysis tools like Meta’s UPM to allow refactoring of tables / DAGs (keep interfaces the same but ‘flatter’ DAGs, less duplicate transforms).

I get what you're saying, but think about a large org with a lot of different teams and heterogenous data stores - it's gonna be pretty hard to implement a top-down directive to tell everyone to use such and such ORM library, or to ensure a common level of ownership and craft. This is where SQL is the lingua franca and usually the native language of the data stores themselves and is a common factor between most/all of them. This is also where tools like Trino / PrestoSQL can come in and provide a compatibility layer at the SQL level while also providing really nice features such as being able to do joins across different kinds of data stores / query optimization / caching / access control / compute resource allocation.

In general it's hard to get things to flow "top down" in larger orgs, so it's better to address as much as you can from the bottom up. This includes things like domain models - it's gonna be tough to get everyone to accept a single domain model because different teams have different levels of focus and granularity as they zoom into specific subsets so they will tend to interpret the data in their own ways. That's not to say any of them are wrong, there's a reason why that whole data lake concept of "store raw unstructured data" came in where the consumer enforces a schema on read. This gives them the power to look at the data from their own perspective and interpretation. The more interpretation and assumptions you bake into the data before it reaches the consumers, the more problems you tend to run into.

That's not to say that you can't have a shared domain model between different teams. There are unsurprisingly also products out there that provide the enterprise the capability to collaboratively define and refine shared domain models, which can then be used as a lens/schema to look at the data. Crucially the domain model may shift over time, so this decoupling of the domain model from the actual schema of the stored data allows for the domain model to evolve over time without having to go back and fix the stored data because we have not baked in any assumptions or interpretations into the stored data itself.

smif

3 jaar geledendiscuss

> Could you share more specific details? Happy to look over / revise where needed.

Sure thing! I'd say first off, the solutions may look different for a small company/startup vs. a large enterprise. It can help if you explain the scale at which you are solving for.

On the enterprise side of things, they tend to buy solutions rather than build them in-house. Things like Informatica, Talend, etc. are common for large enterprises whose primary products are not data or software related. They just don't have the will, expertise, or the capital to invest in building and maintaining these solutions in-house so they just buy them off the shelf. On the surface, these are very expensive products, but even in the face of that it can still make sense for large enterprises in terms of the bottom line to buy rather than build.

For startups and smaller companies, have you looked at something like `dbt` (https://github.com/dbt-labs/dbt-core) ? I understand the desire to write some code, but often times there are already existing solutions for the problems you might be encountering.

ORM's should typically only exist on the consumer-side of the equation, if at all. A lot of business intelligence / business analysts are just going to use tools like Tableau and hook up to the data warehouse via a connector to visualize their data. You might have some consumers that are more sophisticated and may want to write some custom post-processing or aggregation code, and they could certainly use ORM's if they choose, but it isn't something you should enforce on them because it's a poor place to validate data since as mentioned there are different ways/tools to access the data and not all of them are going to go through your python SDK.

Indeed in a large enough company, you are going to have producers and consumers that are going to use different tools and programming languages, so it's a little bit presumptuous to write an SDK in python there.

Another thing to talk about, and this probably mostly applies to larger companies - have you looked at an architecture like a distributed data mesh (https://martinfowler.com/articles/data-mesh-principles.html)? This might be something to bring to the CTO more than try to push for yourself, but it can completely change the landscape of what you are doing.

> More broadly is the issue of the gap of what you think the role is, and what the role actually is when you join. There are definitely cases where this is accidental. The best way I can think of to close the gap is to maybe do a short-term contract, but may be challenging to do under time constraints etc.

Yeah this definitely sucks and it's not an enviable position to be in. I guess you have a choice to look for another job or try to stick it out with the company that did this to you. It's possible there is a geniune existential crisis for the company and a good reason why they did the bait-and-switch. Maybe it pays to stay, especially if you have equity in the company. On the other hand, it could also be the case that it is the result of questionable practices at the company. It's hard to make that call.

smif

3 jaar geledendiscuss

It seems like the article is written by someone just starting to get into the data engineering subfield and they thought they were going to be writing python (pyspark is my guess) to support some kind of ML effort, but they got saddled with a bunch of SQL/data warehousing stuff to support business intelligence/analytics instead. I'd say normally what you say makes sense especially when you're pulling in abbreviations that are not related to the topic at hand or you're introducing new people to the field, but ETL is a pretty basic concept in data engineering and it's a web search away (should be the top result), so I'm not sure if it would really add all that much to their article to start with definitions.

It sounds to me like the author got thrown to the wolves in an environment of what data engineering looked like before "big data" and ML took off (and before it was even really called data engineering). There are a lot of enterprises that are still working in this mode because they are not Google and they don't have the same level of sophistication and automation when it comes to this stuff.

There is some bad information no doubt in the article, but if we're being charitable, it feels like it's someone who took a wrong turn somewhere and is struggling to find their feet in an unfamiliar place without the proper guidance and mentorship and that's a bit admirable at least that they're trying on their own.

There is no direct bearing on ETL in the article, aside from the focus on SQL queries and data validation hints that they might be talking about ELT (Extract-Load-Transform) as the level beyond ETL, but it's not clearly explained. It's clear to me that they are at the start of their journey and they are gonna learn things the hard way without guidance from someone more experienced.

smif

3 jaar geledendiscuss

I'm not saying that hardware shouldn't be open in some form or another (at least in a way that doesn't stifle innovation, maybe also taking another look at how the patent system works and such).

I guess I'm just having trouble visualizing how this problem gets solved for the average joe consumer in a world where hypothetically the hardware is open. Who pushes the security patches out to the devices? All of that has a cost in terms of bandwidth, maintenance, etc. If it's a community effort, what happens when the device gets old enough where no one is really working on it anymore, no more community updates, people have moved on, etc. How does liability work in a world with community-driven updates? What happens if a buggy community update is pushed and the smart fridge malfunctions and causes a flood / damages? What about supply-chain attacks and such?

I guess for the code-literate subset of consumers, they can just go to the github repo and see exactly what is changing and where, but for the non-code-literate consumers, how do they know what kind of updates they are getting from the community?

What about a middle-ground option? Where towards the end-of-life for the product, you are asked a question if you want to switch to a different update channel than the manufacturer default, and if there is no response recorded after X amount of days or whatever, the device just bricks itself?

smif

3 jaar geledendiscuss

How would opening up the hardware solve the issue for the average consumer? Let's say the official update channel goes dead on your smart fridge, the company has gone out of business. What would happen in that scenario for the average consumer (not someone who posesses the skills or will to tinker around with the firmware and such)?

smif

3 jaar geledendiscuss

I think I would agree with pretty much all of the above. There is a sort of hierarchy of belief, and quantum mechanics is probably close to the top of that. However, even so, it is still not infallible. It is possible for us to discover a regime where it breaks down and we need a new theory to supplant it.

This is basically what I'm arguing - no matter how accurately our theories line up with observation, we can never be sure that we have reached "the final theory" AKA the truth. I think this is where a lot of misunderstanding and mistrust for science originates. It will never deliver to us the truth - if it did, how would we ever know?

It is a method of getting closer and closer to what we believe is the truth. But there is still a gap there, however small it might be in the case of quantum mechanics. The scientific method by it's very construction is unable to bridge that gap.

Still, to date there does not exist a more effective method we know of as a species at getting closer to what we believe to be the truth. I think the above is a maybe subtle distinction there that is worth pointing out and educating people on. Just sort of making that distinction between the process and the results. That it is the best process we have, but even so, it cannot cross that gap and definitively say "this is the truth". That that is a gap we have to choose whether to cross ourselves with a leap of faith (or sometimes a very tiny hop of faith in the case of quantum mechanics). I think that might help people cement their faith in the process even if they dont necessarily place their faith in the results (in the case of questionable psych studies for example).

smif

3 jaar geledendiscuss

I wouldn't reject objective facts, but I also wouldn't believe they exist any more than I would believe Santa Claus exists, unless someone can successfully argue for their existence. AFAIK this has yet to be done by philosophers (though there have been many attempts).

E: I should mention that it's not just a binary yes or no here, there is a 3rd option of "I don't know" and I would rabidly defend the "I don't know" camp until someone can convincingly argue one way or the other. All of this has nothing to do with the actual usefulness of science which is unquestionable in my opinion.

This is strictly talking about science "overreaching" into the philosophical realm if you will, where even from the start, methodologically it doesn't have the right tools to answer these questions. You don't prove scientific theories "true", you just accumulate more and more supporting evidence. It never hits a magical moment where the neon lights turn on and a sign says "Your theory is now true! Congratulations!". And even if it did, it would be fleeting anyway because there are no sacred cows here - your theory can just as easily get supplanted by a better theory in light of more evidence.

smif

3 jaar geledendiscuss

If objective reality exists, which is still a pretty big if last time I checked, not only do we not know what it fully is, we don't even know what any part of it is. The best that we can do is get better and better at modeling it in ways that are useful to us (which is what science is doing for us).

smif

3 jaar geledendiscuss

Yes it's a bold claim philosophically. How would you justify it?

No, flat earth "theory", if you can call it that, has close to zero supporting evidence and AFAIK has no actual predictive power. Stick with consensus science if you want actually useful theories, but that is very different from claiming they are giving you objective truth.

Let me ask you this, when a theory that was previously accepted as consensus science loses support in light of new evidence and gets supplanted by a new and better theory, does that mean that the objective truth changed?

smif

3 jaar geledendiscuss

> There are objective facts about the nature of reality

This is a pretty bold claim and you would have to do a bit of work to make it more convincing. Besides, it's not really how science works. Different theories wax and wane over time all the time. What we're really doing with science is coming up with ever better models that give us greater predictive power.

You could argue that at the macro scale we're asymptotically approaching some kind of objective truth with the whole endeavor of science, but you can't simply tunnel across that gap and make the leap to say that we know there is an objective truth.

The best that we can do is probably say something along the lines of "these are the best methods of getting closer to the truth that we have available - if anyone claims to have better methods, they are very likely wrong", but you still need to have the humility to accept that even the best models that we have to date are not infallible. They do not give us the objective truth, nor do they promise to, but they are the most effective tools that we have available to us at the current time. This is critically not the same as claiming that they give us the objective truth.

You could say that for all intents and purposes/everyday usage, "sure, these are objective facts about reality" - but I don't actually think that helps anyone and it serves to foment mistrust towards science and scientists. I think the best chance we have at preserving the status of science and scientists in our society starts by being honest about what it actually is giving us - which is quite a lot, but let's not oversell it for the sake of convience or whatever.

smif

3 jaar geledendiscuss

I feel like that's where most people probably start, and then get some response like "we can't change the KPI's because blah blah" or get the runaround. Once you have established that you have no agency to change the system is when you are sort of incentivized to let it fail in hopes that a better system replaces it one day. You also don't wanna piss too many people off that may be invested in the current system because you want those references for your next job.

smif

3 jaar geledendiscuss

Not everyone, I still do not have this in mine and I have paternal haplogroup G-M201 which is related to Ötzi.

smif

3 jaar geledendiscuss

The paper seems to be paywalled so I have no clue about how they arrived here, but this: "That said, the results indicate that the probability we are alone (<1) in the galaxy is significant, while the maximum number of contemporary civilizations might be as few as a thousand."

Doesn't seem to really answer anything. Isn't this just a really fancy way of saying "we don't know the solution to the Drake equation"? It could be a 99.999999...% (<100%) chance of being alone, it could be a 0% chance, or anything in between.

Given the title of the paper, this is a very loose definition of "solution" for the Drake equation ("it could be anything!").

smif

3 jaar geledendiscuss

What about a third option, "everything may or may not matter, but the answer to that question is currently inaccessible to us (and possibly may always remain inaccessible)"?

In this way you could be led to a kind of inverted Pascal's wager, where you can't reasonably go down the nihilist route because everything might just might matter, but you just don't know. You also don't know in which ways it might matter if it does, so you don't really have a conclusion to draw about where to go from here.

smif

3 jaar geledendiscuss

What lay people mostly mean by 'relative' is subjective; ie., they're committing the genetic fallacy of the form: since understanding X requires a perspective in which to evaluate it, X itself must be a (inter)subjective phenomenon.

ie., they're saying that since understanding is relative to a perspective so is what is understood. This is the irritating fallacy popular amongst those who go around abusing the term 'relative'.

I'm not sure if I find this particularly convincing. I think most lay people are actually talking about the thing itself rather than their understanding/perspective on the thing.

Maybe this is something that a study could shed light on, if we're able to construct a survey that can give us some insight into how lay people are using/abusing these terms.

smif

3 jaar geledendiscuss

I've also found talking about what you just learned is very helpful. If you don't have an interlocutor available, ChatGPT is a good stand-in.

smif

3 jaar geledendiscuss

> It's exactly the other way around. It's lay people using 'relative' to mean 'subjective'.

I've actually found the opposite - people will often use "subjective" when they really mean "relative". They often unwittingly imply the relation to culture. I think it's far more rare to find a lay person using the term "subjective" to precisely refer to the philosophical meaning of the word.

> And lay people use 'objective' to mean 'universal', which it doesn't. Cultural relativism is consistent with an objective metaethics.

It's true there is a distinction there between "objective" and "universal", but do you feel that in most contexts, when someone says "objective" they are also implying "universal" and the onus is on them to elaborate on the distinction when that implication doesn't hold? (I guess asking for both the case of lay people and the one for philosophers).

comments