daamien

OP is one of 7 PostgreSQL core team members. That's kind of his job to massively overthink this :)

Hi !

I'm the main developer of this extension. Happy to answer any question you have about this project and anonymization in general!

daamien

2년 전discuss

Pseudonymizing functions are just one way to mask the data.

There are many other masking functions that will actually anonymize the data.

And the extension does not force you to respect the foreign keys.

It's really up to you to decide how you want to implement your masking policy

daamien

2년 전discuss

PostgreSQL Anonymizer developer here : this was possible with version 0.8 which is now completely unmaintained. This extension is currently not available on RDS but it is available on many others DBaaS providers : Azure SQL, Google Cloud SQL, Crunchy Bridge, ....

daamien

2년 전discuss

Both approach are possible

* Static Masking will destroy the authentic data once for all

* Dynamic Masking will only alter the data the "masked users". Regular users will continue to view the real data.

Both techniques have their own advantage depending on your context.

daamien

2년 전discuss

This is called "Privacy By Default"

https://postgresql-anonymizer.readthedocs.io/en/stable/priva...

daamien

2년 전discuss

Of course there's no perfect solution for anonymizing a dataset...

The extension offers a large panel of masking functions : some are pseudonymizing functions but others are more destructive. For instance there's large collection of fake data generators ( names, address, phones, etc. )

It's up to the database administrator or the application developer to decide which columns need to be masked and how it should be masked.

In some use cases, pseudonymization is enough and others anonymization is required....

daamien

2년 전discuss

Version 2.0 was released a few days ago. It's a complete rewrite in Rust.

Performance should be better than with v1.x

daamien

2년 전discuss

This called "Masking Data Wrappers"

https://postgresql-anonymizer.readthedocs.io/en/stable/maski...

daamien

2년 전discuss

PostgreSQL Anonymizer developer here : this was possible with version 0.8 which is now completely unmaintained.

This extension is currently not available on RDS but it is available on many others DBaaS providers : Azure SQL, Google Cloud SQL, Crunchy Bridge, ....

daamien

2년 전discuss

PostgreSQL Anonymizer developer here : the problem you are describing here only affects version 1.x

Version 2.0 was released a few days ago with a branch new masking system that does not block database migrations.

daamien

3년 전discuss

Actually the GDPR "just" requires to protect the data against "reasonably likely" attacks.

« To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly.

To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments. »

https://www.privacy-regulation.eu/en/r26.htm

daamien

7년 전discuss

See PEV2 : https://github.com/dalibo/pev2

daamien

8년 전discuss

This is still an issue :

https://postgresql.verite.pro/blog/2018/08/27/glibc-upgrade....

daamien

8년 전discuss

... and with dozens of other data sources:

https://wiki.postgresql.org/wiki/Foreign_data_wrappers

daamien

8년 전discuss

PostgreSQL does this out of the box with :

  CREATE EXTENSION file_fdw;
  CREATE SERVER import FOREIGN DATA WRAPPER file_fdw;
  CREATE FOREIGN TABLE foo (
    col1 text,
    col2 text,
    ...
  ) SERVER import 
  OPTIONS ( filename '/path/to/foo.csv', format 'csv' );
  SELECT col1 FROM foo WHERE col2='x';

daamien

10년 전discuss

direct link to the PDF : http://www.postgresql.org/message-id/attachment/43080/Postgr...

daamien

11년 전discuss

Temporary index and Hypothetical index are two different things.

Temporary index is a real index that's usually dynamically created by the optimizer to speed up a query. But it's kind of useless because most of the time the cost of creating a one-time index exceed the cost of not using it. Afaik only DB2 as this feature

http://www.centerfieldtechnology.com/PDFs/DB2%20Temp%20Index...

An hypotetical index is just a way to "trick" the planner into thinking that there's an index and see what the query plan would be if that index really existed. The cost of creating this is close to zero.

daamien

12년 전discuss

Hi !

I'm involved in both projects. Here' s some insight :

OPM and POWA are not related. They were born at different times and developed by two different R&D teams at DALIBO, a French PostgreSQL company. The fact that they've been released publicly almost at the same time is merely a coincidence

OPM is designed to help a DBA to manage a very large number of servers. It provides answers to the questions : "Is there a problem somewhere on my PostgreSQL servers ? Which instances need to improvements ?"

POWA is more focused showing the "real-time" activity of one specific instance. It provides answers to questions like : "Which queries are slowing down my server right now ?"

If you're familiar with Oracle products, the difference between OPM and POWA is more-or-less the difference between Oracle AWR and Oracle Grid Control.

In a nutshell, if you only have one or two PostgreSQL servers in production, then POWA is easier to install and probably enough for you. If you want more stats and an overview of your servers, then OPM is a better choice.

Both projects are autonomous and moving fast. We're just pushing code on github without direct economic goals or marketing plans :-) In the long run we'll see which one finds its user base and how they evolve.

That being said, we also have other guys working on other similar PostgreSQL tools, in particular :

pgBadger ( http://dalibo.github.io/pgbadger/ ) is a log analyzer which provides rich reports of the database activity. Basically it answers to the question "Which queries should be optimized ?"

pgCluu ( http://pgcluu.darold.net/ ) is a audit/reporting tools, more focused on system and resource utilization. It answers to questions such as "What's the load on my CPUs ? Is my memory swapping ?"

pg_activity ( https://github.com/julmon/pg_activity/ ): is like linux "top" command but for PostgreSQL queries.

pg_stat_kcache ( https://github.com/dalibo/pg_stat_kcache ) will gather statistics about reads done by the filesystem layer

So basically regarding PostgreSQL monitoring, we're just moving in every possible directions at the same time... Some say it's a "waste of time", we prefer the term "Darwinian Approach" :P

daamien

12년 전discuss

fixed.

Submissions

The strange case of the underestimated Merge Join node

Pipeline Mode in Psycopg

How Psycopg row factories help you write safer applications

Again Riddle

Oracle's letter to Russian IT companies

Spotify PostgreSQL Metrics

PostgreSQL 9.4.1 Released

A new vmstat-like tool for PostgreSQL

PostgreSQL Benchmark On Amazon EC2 And Rackspace

Mongo + Postgres = Mongres

French government outlines plans for free software adoption

comments