zombodb · HackerTrans

ปีที่แล้วdiscuss

Shout out to grapeseed oil

2 ปีที่แล้วdiscuss

Yup. Just under a 501c3 to help ensure pgrx outlives us.

Huge thanks to TCDI (www.tcdi.com) for continuing to fund development.

zombodb

2 ปีที่แล้วdiscuss

Y'all have built something cool. I'm excited to see where it goes from here.

If there's anything the pgrx team can do to help, just let us know.

zombodb

2 ปีที่แล้วdiscuss

If you build it, they'll come. The only limit is yourself!

zombodb

3 ปีที่แล้วdiscuss

Nice deck, Ryan! And thanks for the mentions. ;)

zombodb

3 ปีที่แล้วdiscuss

I think maybe what you’re really looking for are the files here: https://github.com/pgcentralfoundation/pgrx/tree/c2eac033856...

Those are the internals we currently expose as unsafe “sys” bindings.

As we/contributors identify more that are desired we add them.

pgrx’ focus is on providing safe wrappers and general interfaces to the Postgres internals, which is the bulk of our work and is what will take many years.

As unsafe bindings go, we could just expose everything, and likely eventually will. There’s just some practical management concerns around doing that without a better namespace organization —- something we’ve been working.

The Postgres sources are not small. They are very complex, inconsistent in places, and often follow patterns that are specific to Postgres and not easy to generalize.

If you’ve never built an extension with pgrx, give it a shot one afternoon. It’s very exciting to see your own code running in your database.

zombodb

3 ปีที่แล้วdiscuss

I don’t think we even expose the TableAM APIs? They are incredibly hard to generate bindings for from the C headers — lots of inline functions and complex #define macros.

We have an ambitious goal with pgrx and it’s going to take many years and countless hours of developer effort to get there.

It can, however, serve as a way for newcomers to gain experience with Postgres internals.

Regarding the TableAM specifically, when we are able to create a safe Rust wrapper around it, that wrapper will be documented.

zombodb

3 ปีที่แล้วdiscuss

Thank you. I’ll pass this on to the team.

zombodb

3 ปีที่แล้วdiscuss

If you’re aware of bugs that fall within plrust’s domain we’d appreciate any and all reports via GitHub!

zombodb

4 ปีที่แล้วdiscuss

Always nice to see pgx being used in the wild! Awesome work.

zombodb

5 ปีที่แล้วdiscuss

I don’t pay any attention to what the cloud providers are doing. I have no control over them so….. eh.

ZDB is still alive and well but I have a real job now too.

zombodb

5 ปีที่แล้วdiscuss

I can definitely reword that if it's confusing.

zombodb

5 ปีที่แล้วdiscuss

> Doing so would require ElasticSearch to reach consensus on every read/write

ZomboDB only requires that ES have a view of its index that's consistent with the active Postgres transaction snapshot. ZDB handles this by ensuring that the ES index is fully refreshed after writes.

This doesn't necessarily make ZDB great for high-update loads, but that's not ZDB's target usage.

> They also claim that transactions will abort if ElasticSearch runs into network trouble...

I had to search my own repo to see where I make this claim. I don't. I do note that network failures between PG & ES will cause the active Postgres xact to abort. On top of that, any error that ES is capable of reporting back to the client will cause the PG xact to abort -- ensuring consistency between the two.

Because the ES index is properly refreshed as it relates to the active Postgres transaction, all of ES' aggregate search functions are capable of providing proper MVCC-correct results, using the parallelism provided by the ES cluster.

I don't have the time to detail everything that ZDB does to project Postgres xact snapshots on top of ES, but the above two points are the easy ones to solve.

zombodb

5 ปีที่แล้วdiscuss

I wonder what the ZomboDB developers are up to now? What great text-search-in-postgres things could they be secretly working on?

zombodb

6 ปีที่แล้วdiscuss

https://github.com/zombodb/pgx/releases/tag/v0.0.7

!boom!

zombodb

6 ปีที่แล้วdiscuss

I'd be inclined to agree.

That said, it'd be easy to prototype with pgx, assuming that compression library exists in the Rust ecosystem too?

zombodb

6 ปีที่แล้วdiscuss

I've noodled the idea of a pl/pg_rust, but the whole "Cargo.toml" and downloading dependencies from the internet seems bad.

I see what you're saying about no_std tho. Could be doable.

zombodb

6 ปีที่แล้วdiscuss

That is interesting.

pgx does a lot of that too, but for only pg10/11/12. And then it post-processes the bindings and builds a "common.rs" for symbols that are identical across versions.

That way you can target all three versions at once, and then you can use Rust's #[cfg(feature=)] directive to opt into support for specific versions.

pgx has also done a little bit of work to mask differences between the three versions so they appear as part of "common.rs" anyways.

I wanted a framework that let me know, at development time, if I was trying to do a thing that isn't supported (or the same) across various PG versions.

zombodb

6 ปีที่แล้วdiscuss

blush

zombodb

6 ปีที่แล้วdiscuss

Sure! Top is Postgres, bottom is pgx, after running each 5 times...

    test=# SELECT count(*) FROM (SELECT generate_series(1, 10000000)) s;
      count   
    ----------
     10000000
    (1 row)

    Time: 399.630 ms
    test=# SELECT count(*) FROM (SELECT srf.generate_series(1, 10000000)) s;
      count   
    ----------
     10000000
    (1 row)

    Time: 478.194 ms

Thanks for the reply. I'm not surprised there's room for optimization in pgx, especially in the Iterator-->SRF path.

edit: PS, thanks for the idea of putting what I assume is the backend pid in psql's prompt. I need to go figure out how to do that right now!

edit edit: hmm, I guess that's not the PID.

comments