Engineering hasn't gone away, you're now just directing things at a higher level. You are now a architect & manager (but you're managing agents not people).
Who sometimes has to deep dive & mentor a agent on solving the right problem.
I'm not sure I would want this even if I could have it TBH. Engingeering org size is about ~200 with infra/sre/ops around ~25.
Different teams want to move at difference cadences. At a certain scale splitting up things feels a little more natural (maybe I am stockholmed by prior limitations with TF though or just used to this way of operating now).
But even then, we're moving to k8s operators to orchestrate a bunch of things and moving off terraform apart from the stuff that doesn't change much (which will eventually get retired as well). Something like https://www.youtube.com/watch?v=q_-wnp9wRX0
Terraform variable management is our larger problem (now/nearterm) when we have to deploy numerous cells of infra that use the same project/TF files with different variables. Given the number of projects/layers of TF getting cell specific variables injected is meh.
Those variables are instance size, volume size, addresses, IAM policy, keys etc.
This is in the b2b saas world with over a million MAU. We've got islands of infra for data soverignty, some global cells where each cell can communicate back / host some shared services (internal data analytics, orchestration tooling, internal management tooling and the like).
This is awesome, I will be trying this out in the coming months. Its just made it to the top of my R&D shortlist for things that could massively simplify our data stack for a b2b saas.
It looks like people are using it to build graph related models on it.
I am looking at it & considering doing something similar for graph data sets. As well as a transactionally safe key value store to store roaring bitmaps.
We had to stop & wait at various points on our return day trip for other passenger services to pass us. Had to wait for some freight services to clear a block at a few places too.
There was an auxiliary diesel unit not actively used apart from for power AFAIK (and additional emergency mobility should things go kaboom/excess wheel slip).
Maybe that has some the extra safety systems on it though.
I started looking into podman quadlets on the weekend. All because docker & nft (nftables firewall) don't play nice together.
I have my finely crafted nftables ruleset that I want to keep. Docker is going to abuse the crap out of my ruleset/make me jump through extra hoops to have it work.
Having the containers operate under systemd looks fine (ignoring the fact that it is systemd, I guess I have finally accepted systemd is here to stay).
I was interested in using traefik & docker labelling to have ingress sorted out and I believe this will all play nicely enough to replace docker-compose on a single server.
https://youtu.be/qcNNB2qZ5kE?t=2776 around this time code. If the USA doesn't want a nation to fight then there are more issues than a F35 software kill switch etc.
Just starting to review it but my front of mind questions:
1) How do I handle persistence? Looks like some code is missing.
2) Do you support multi-tenancy (b2b saas graph backend for handling relations scoped to a tenant)
Interesting in hearing some thoughts about using roaring bitmaps stored in a bytea postgres column to represent adjacency matrixes.
I was thinking that given RDS has support for plrust and PostgreSQL's SPI I could use the fact they support croaring-rs there as a crate and build upon that.
I figure I can use that to represent many graph's with say 100s to ~100m nodes and many relations between these things. But each graph would be tenanted to a tenant (company/b2b saas use case).
I was thinking that by using plrust storing the roaring bitmap on the DB server in a bytea and using SPI, I can benefit from the minimal network overhead to mutate and query against the bitmap with croaring. Using SPI locally in the DB server I eliminate network overhead shipping that back to my application code.
PostgreSQL also gives me transaction safety to updates etc. And a bunch of support for other column base data such as my tenant ID column, some JSONB for relationship metadata to query on etc.
Basically something like https://jazco.dev/2024/04/20/roaring-bitmaps/ but on postgres. Given I need to support many tenanted graphs & we're already using citus this seems like something that is feasible at a larger scale too.
I was wondering though if I am going to need to create some operator classes to allow me to index relations a bit better (probably seems likely I think).
I am aware of https://github.com/ChenHuajun/pg_roaringbitmap but would prefer to use int64s and maybe start out on RDS instead of having to add another workload to our self hosted citus cluster/s.
Happy to be told I am fool and any insights would be nice. I am potentially going to try this out on some of our data sets we have because our product team is basically laying out a vision where they want us to have a graph powering a bunch of things.
I don't like the idea of neo4j when we're already deep into PostgreSQL for a bunch of workloads (~20+ TB table workloads etc so we have some reasonable inhouse PG experience).
Also huge thanks to the author of the blog post. I had been looking at pgRouting and wondering with a tilted head.. hmm seems like we can just use this as a graph DB. So that is also on my list to test out.
They work with sulphuric acid, oxy/acetylene/propane torches.
Some processes give off cyanide fumes.
Ultrasonic baths with ammonia based solutions to clean polishing compounds off etc.
You need outside air ventilation.
There is lots of mechanical suction for things like polishing to capture the waste material for post processing.
Most will wear a leather apron for heat / burn protection and capturing fine dust/dirt from polishing compounds. I suppose you could destroy that eventually in a giant smelter.
I did my jewellery trade in Australia (hence the correct spelling for me). We used to keep all our emery paper, old polishing wheels etc and send them off ever few years to be burnt & refined.
When the building we were in got renovated some enterprising guys in another workshop ripped up their floor boards and their neighbouring empty suites and got all the precious metals out of the gaps between the floorboards.
The building was 11 stories and was predominantly filled with small jewellery workshops with 2-5 people per business. And a lot of adjacent businesses (trade supplies, stone merchants etc).
I'm not a frontend engineer or really a fully experienced backend engineer. I have done systems stuff, lots of OPS, scaling and other things adjacent.
Flutter is interesting to me because I can use a tool like flutterflow to build an app and throw together the backend reasonably comfortably (with some possible future footguns included).
Not knowing Kotlin means this is less attractive to me, but I can see the benefit if I knew one language that could span frontend and backend very nicely.
Laughs at a PostgreSQL schema change backfill jobs that need a few days to complete on i4i.16xlarge.
Or that time I spent $120k USD on testing some upgrade paths on a dataset in elasticsearch from version 2.4 -> 5 -> 6 -> 7 that took about 5-6 days runtime on imports per upgrade step.
Do you have billions of rows with only a very small % in use/locked or millions of rows that are jsonb blobs with a relatively high % in use/locked?
Is your workload mostly write once read lots or read/write evenly split etc.
How fast is your IO. Are you using network storage (EBS for example) vs local NVMe?
How much other load are you contending with on the system.
I have a JSONB heavy workload with lots of updates. JSONB blobs avg around 100KB. But can go up to 20MB. We can see < 10 updates on a blob all the way to thousands of updates on the larger ones.
We use Citus for this workload and can move shards around and that operation will use logical replication to another host effectively cleaning up the bloat that way.
We also have some wide multi-column indexes over text fields and date fields that see a fair bit of rewrite activity as well. Those indexes get bloated a fair bit too and we run re-indexes every few months (used to be every 6-12 months, but as the workload is getting busier we're re-indexing a bit more frequently now).
The index bloat is far more of a problem than the raw table bloat.
In the past I would use pg_repack when we were on a single RDS solution.
I have a machine, took a few weeks to dial in the settings right for me. Tried a few different masks & machines from a sleep study clinic. In a few years I have gone less than 7 nights without using it.
Using the machine has been life changing for me.
I used to struggle to get up before 10am. Now I wake up naturally around 6:30am-7am most mornings.
Learning new things isn't a mountain to climb to retain just a bit of information. I feel like if I had done a sleep study 10 years earlier I would be a much better technologist than I am now.
I would recommend you suggest to your wife that she tries a few different mask styles.
I initially found I wasn't fitting the mask well or had a slightly wrong size and I would get some leaking that would keep me awake at night.
Taxi's in NSW/Sydney are much better these days. Uber X are typically shit by comparison.
A $95 UberX ride home is about $109 in a taxi for me.
Talking to a few friends who also live a similar distant from the CBD/downtown find that Taxi's are the better / more reliable option these days too.
The taxi is pretty much always newer, with a professional driver who usually isn't almost falling asleep behind the wheel. The quality of the driving is typically significantly higher.
It feels like a safer ride.
Of the recent UberX's I have caught, the car was typically in need of a suspension replacement or significant servicing and around 6-7 years old or older.
Most recent taxi rides I have had a car around 12 months old or less.
Taxi's are better regulated and provide a better service. I'll happily pay the little premium there is in this market to use one over an Uber.
On a recent trip to NZ the uber to taxi price comparison was massively different.. Taxi was like 2.5x more expensive..
Who sometimes has to deep dive & mentor a agent on solving the right problem.