The Functional Database [video](infoq.com)
infoq.com
The Functional Database [video]
http://www.infoq.com/presentations/datomic-functional-database
8 comments
Such a shame about InfoQ. Seems like they've got lots of interesting content, but the user experience is abysmal to the point where I now avoid the site. Would it be so bad to just let us flick through the slides?
When I'm in a rush for a tldr, I want to flick through a few slides and move on with browsing something else. But, there are a lot of times that I'm not in a rush. When I'm taking a walk or riding the bus, an MP3 of a presentation is a great way to have someone slowly explain an idea that I'm interested in -but not enough to actively concentrate on researching.
Relaxing at home on the couch is another time that I'm not in a rush. I've largely replaced evening TV time with watching presentations on various topics. InfoQ is perfect for that. I get to have an expert take an hour to carefully explain a topic that I'm only tangentially interested in while I semi-nap on my couch after dinner with a beer. This has become a low-effort means for me to gain a high-level understanding of a broad range of topics that I wouldn't otherwise have the time or willpower to sit down and actively study.
Relaxing at home on the couch is another time that I'm not in a rush. I've largely replaced evening TV time with watching presentations on various topics. InfoQ is perfect for that. I get to have an expert take an hour to carefully explain a topic that I'm only tangentially interested in while I semi-nap on my couch after dinner with a beer. This has become a low-effort means for me to gain a high-level understanding of a broad range of topics that I wouldn't otherwise have the time or willpower to sit down and actively study.
if you create an account on infoq, you're able to download the slides.
Everything Rich says and does is gold!
It surprises and disappoints me that he is not as well known nor revered as the agile craftsmanship peddlers.
It surprises and disappoints me that he is not as well known nor revered as the agile craftsmanship peddlers.
That's because agile provides hype to everyone in the company, not just developers. Every single agile consultant I've met was a mediocre programmer at best.
What rich says, on the other hand, is pure gold as you said. But it needs to be understood before its value is seen.
It's hard to sell to the manager who has no clue what all of this means, even harder to compare the tradeoffs with their current approach.
What rich says, on the other hand, is pure gold as you said. But it needs to be understood before its value is seen.
It's hard to sell to the manager who has no clue what all of this means, even harder to compare the tradeoffs with their current approach.
I know very well who Rich is, but I don't know (and don't want to know) any of the "agile" proponents.
Early on in the movement, one of the big names who also had some fairly close corporate associations came to speak at a user group meeting hosted at my then BigCo location. I even passed news of the event around so that interested parties in our local shop could head down after hours and have a listen.
I felt rather bad afterwards for having done so. One of the more useless talks I've attended. Very superficially prescriptive (you should do X; Y has seen benefit Z), with no meat to the description (or prescription, as it were).
Agile may well have its place. But its intersection with BigCo life, as anecdotally exemplified in this fellow, seemed to be just more of the "same old same old". New words, same shit.
I felt rather bad afterwards for having done so. One of the more useless talks I've attended. Very superficially prescriptive (you should do X; Y has seen benefit Z), with no meat to the description (or prescription, as it were).
Agile may well have its place. But its intersection with BigCo life, as anecdotally exemplified in this fellow, seemed to be just more of the "same old same old". New words, same shit.
> It surprises and disappoints me that he is not as well known nor revered as the agile craftsmanship peddlers.
Actually, he is better known than all those guys selling snake oil.
Actually, he is better known than all those guys selling snake oil.
Can someone provide a one paragraph definition of the functional database?
Is three ok? Using my limited experience playing with Datomic:
A functional DB replaces a "row" that only holds the current state of the world with a list of timestamped "facts" that can represent either the present or any point in the past.
Instead of simply recording "Bob's favorite food is pizza," we record that "'Bob's favorite food is pizza' was added at Time X." If Bob decides his favorite food is now ice cream, we record that "'Bob's favorite food is ice cream' was added at Time Y." At any time, we can query Datomic to get the current state of the world--"Bob's favorite food is ice cream." But note that we can also query Datomic to get any past state of the world. If our query is "what was Bob's favorite food 10 minutes before Time Y?" the response will be "pizza."
A regular database is like a sheet of paper that is continually erased and drawn over with a new world-state. Datomic, as a "functional database," is like a flipbook with an updated state on every page.
A functional DB replaces a "row" that only holds the current state of the world with a list of timestamped "facts" that can represent either the present or any point in the past.
Instead of simply recording "Bob's favorite food is pizza," we record that "'Bob's favorite food is pizza' was added at Time X." If Bob decides his favorite food is now ice cream, we record that "'Bob's favorite food is ice cream' was added at Time Y." At any time, we can query Datomic to get the current state of the world--"Bob's favorite food is ice cream." But note that we can also query Datomic to get any past state of the world. If our query is "what was Bob's favorite food 10 minutes before Time Y?" the response will be "pizza."
A regular database is like a sheet of paper that is continually erased and drawn over with a new world-state. Datomic, as a "functional database," is like a flipbook with an updated state on every page.
Okay, so it's a database with version control. I was actually looking for a feature like this a while back in normal databases.
It's more than that. It's actually a lot like distributed version control (a la git). When you ask the connection for the current value of the database, you get back a real value that will never change out from under you. This is very much akin to making a clone of a repository in git. This gives you the enormous power of speculation and experimentation within your application. You can do whatever you want with this database value and it will never affect anyone else nor be affected by anyone else.
In the database world, we call it temporal databases, because it introduces the concept of time as a first-class part of the conceptual/logical model.
Every production database I've ever seen goes through some version of the same evolutionary lifecycle:
1. "We only need the current state". The database acts as a snapshot of the current world. Updates cause the loss of historical data. The database is like a state machine.
2. "Oops, actually, we need point-in-time reports". The database is hacked with date_from and date_until fields (which introduce interesting anomalies and impose programming overhead on every query written).
2a. "This is a mess, let's clean it up". The database schema is refactored so that the central model is logs of transactions. Point-in-time snapshots are derived at query time. Note that this both replicates the underlying logic of database design (much as network protocol layers are fractal). Note also that it recreates the way basic accounting works, which was the inspiration for database transactions.
3. "Oh crap, the regulations/laws/reporting standards changed". Now you need yet another layer to represent changes in the domain, not just in the data. Your point-in-time reports become even hairier as you must now write different queries depending on the time period being accounted for; and sometimes you must write queries that span both periods and include logic to combine them.
The concept of a temporal database is to make points-in-time a universal, cross-cutting part of everything that happens to the database, either in the schema or the data. Rich has correctly identified the correspondence with functional immutability, where instead of modelling things as having mutable state, you model changes as a series of successor models, each of which is by itself immutable.
I think it's a good idea. The world would be very different if proper temporal logic had been baked into SQL in the first place.
Every production database I've ever seen goes through some version of the same evolutionary lifecycle:
1. "We only need the current state". The database acts as a snapshot of the current world. Updates cause the loss of historical data. The database is like a state machine.
2. "Oops, actually, we need point-in-time reports". The database is hacked with date_from and date_until fields (which introduce interesting anomalies and impose programming overhead on every query written).
2a. "This is a mess, let's clean it up". The database schema is refactored so that the central model is logs of transactions. Point-in-time snapshots are derived at query time. Note that this both replicates the underlying logic of database design (much as network protocol layers are fractal). Note also that it recreates the way basic accounting works, which was the inspiration for database transactions.
3. "Oh crap, the regulations/laws/reporting standards changed". Now you need yet another layer to represent changes in the domain, not just in the data. Your point-in-time reports become even hairier as you must now write different queries depending on the time period being accounted for; and sometimes you must write queries that span both periods and include logic to combine them.
The concept of a temporal database is to make points-in-time a universal, cross-cutting part of everything that happens to the database, either in the schema or the data. Rich has correctly identified the correspondence with functional immutability, where instead of modelling things as having mutable state, you model changes as a series of successor models, each of which is by itself immutable.
I think it's a good idea. The world would be very different if proper temporal logic had been baked into SQL in the first place.
Right the term functional is just confusing. Temporal database theory is relatively mature and even part of the SQL standard for which IBM implements in DB2.
Any database that uses a append only log can potentially be temporal so long as it retains the log and exposes the information in the log (which most don't).
Even say PostgreSQL uses a form short term temporal querying in order to provide MVCC, to bad they haven't taken it further to allow fully temporal standards.
Any database that uses a append only log can potentially be temporal so long as it retains the log and exposes the information in the log (which most don't).
Even say PostgreSQL uses a form short term temporal querying in order to provide MVCC, to bad they haven't taken it further to allow fully temporal standards.
I spent years making all kinds of databases, and I approve your message.
I recently noticed that CouchDB with its notion of map-reduce views can be used as a temporal database - make each change a "document", then make a map-reduce job to roll up the changes to the present state (or "as of" state", if you need it). Nice side effect is that you get free multi-master replication with automatic conflict resolution.
Do you have any other suggestions for a temporal database?
I recently noticed that CouchDB with its notion of map-reduce views can be used as a temporal database - make each change a "document", then make a map-reduce job to roll up the changes to the present state (or "as of" state", if you need it). Nice side effect is that you get free multi-master replication with automatic conflict resolution.
Do you have any other suggestions for a temporal database?
To my very great shame I am aware that there is an entire literature on the topic and I have barely even skimmed the surface of it.
Snodgrass, a leading researcher on temporal databases, wrote a book in 90s about it which is available for free from his website[1].
It's good because it was written before some temporal extensions were added in SQL:2003 (I think). The problem is that almost nobody has implemented those extensions (I believe some versions of DB2 have them), so you are left with doing things by hand. The Snodgrass book goes into amazing detail as to why you'd do such a thing and how to do it.
[1] http://www.cs.arizona.edu/~rts/tdbbook.pdf
Snodgrass, a leading researcher on temporal databases, wrote a book in 90s about it which is available for free from his website[1].
It's good because it was written before some temporal extensions were added in SQL:2003 (I think). The problem is that almost nobody has implemented those extensions (I believe some versions of DB2 have them), so you are left with doing things by hand. The Snodgrass book goes into amazing detail as to why you'd do such a thing and how to do it.
[1] http://www.cs.arizona.edu/~rts/tdbbook.pdf
This should be sent to everyone designing a database for anything more than a quick prototype. I agree completely with your lifecycle :)
Temporal databases and immutability are independent concepts. A 'time series database' only solves the simple temporal database problems. When validity comes into play things become really hard (something was entered yesterday but will be valid from tomorrow).
Temporal databases rely on the immutability of each record (whether by convention or as a system guarantee). Changes are no longer expressed with updates on a given row; instead you add new rows to represent the change.
> Temporal databases rely on the immutability of each record
Some maybe, certainly not all! What you mean are 'historical' databases, a.k.a. time series.
Some maybe, certainly not all! What you mean are 'historical' databases, a.k.a. time series.
> A functional DB replaces a "row" that only holds the current state of the world with a list of timestamped "facts" that can represent either the present or any point in the past.
... which isn't 'functional' in any sense. It seems that Hickey just reinvented time series and tries to sell those as database.
... which isn't 'functional' in any sense. It seems that Hickey just reinvented time series and tries to sell those as database.
> ... which isn't 'functional' in any sense.
Functional programming is very closely tied to the notion of immutable data. "Don't modify data in place" is one of the pillars of the functional model.
> It seems that Hickey just reinvented time series and tries to sell those as database.
Datomic is much more than a time-series, although that's the aspect that arguably makes it "functional." For example, it provides a datalog query interface and has a pretty fascinating and unique architecture that provides extremely high scalability for queries. If you don't like Datomic, or if it doesn't suit your purposes, or if you think it's overpriced for its value-added (I do), that's cool, but don't dismiss it based on a malformed understanding of what it is.
Functional programming is very closely tied to the notion of immutable data. "Don't modify data in place" is one of the pillars of the functional model.
> It seems that Hickey just reinvented time series and tries to sell those as database.
Datomic is much more than a time-series, although that's the aspect that arguably makes it "functional." For example, it provides a datalog query interface and has a pretty fascinating and unique architecture that provides extremely high scalability for queries. If you don't like Datomic, or if it doesn't suit your purposes, or if you think it's overpriced for its value-added (I do), that's cool, but don't dismiss it based on a malformed understanding of what it is.
> Functional programming is very closely tied to the notion of immutable data. "Don't modify data in place" is one of the pillars of the functional model.
I don't think the Lisp fans here at HN agree.
> Datomic is much more than a time-series, although that's the aspect that arguably makes it "functional."
Not really. Time-series are not 'functional' by any stretch of the imagination.
BTW, an add-on to Hibernate, the (in)famous OR-mapper, tries to do something similar: http://docs.jboss.org/envers/docs/index.html
I don't think the Lisp fans here at HN agree.
> Datomic is much more than a time-series, although that's the aspect that arguably makes it "functional."
Not really. Time-series are not 'functional' by any stretch of the imagination.
BTW, an add-on to Hibernate, the (in)famous OR-mapper, tries to do something similar: http://docs.jboss.org/envers/docs/index.html
It's a functional database since it allows you to create database applications using functional programming.
Not quite. A time series' purpose is to record the change in value of some quantity over time, while this ties with the notion of persistent data structures and Hickey's philosophy of value and state which is at the heart of Clojure.
The purpose is not to follow a changing quantity, but to formalize state and value. The idea is that any entity's state, at any given moment in time, is an immutable value. You can query the state at time t0, get a value, and examine it or process it for as long as you like. You can then query it again at time t1, and get another, immutable value.
I guess this is a dual way of looking at time series data, with a different emphasis. The time series is intended to help with processing change, while a persistent data structure is intended to help working with a snapshot of the world as it existed in a specific instant.
The purpose is not to follow a changing quantity, but to formalize state and value. The idea is that any entity's state, at any given moment in time, is an immutable value. You can query the state at time t0, get a value, and examine it or process it for as long as you like. You can then query it again at time t1, and get another, immutable value.
I guess this is a dual way of looking at time series data, with a different emphasis. The time series is intended to help with processing change, while a persistent data structure is intended to help working with a snapshot of the world as it existed in a specific instant.
Doesn't that mean that the database continually grows in size?
It does the same way a git repository does, by accreting deltas with enough metadata to reconstruct the full data at any point in time.
It means that you can write pure functions (in the functional programming sense) where one (or more) of the arguments to your function is a _database value_. In haskell the types to a query function would perhaps look like this:
query :: Query -> DBValue -> Result
performQuery :: Query -> IO DBConn -> IO Result> Summary
> Rich Hickey discusses how a functional database can impact the programming model, using Datomic as an example, but the principles apply to other systems using an immutable database.
It's just an advert of Datomic.
It's just an advert of Datomic.
I expect it to be like this:
(SELECT
("id", "login", "first_name", "last_name")
FROM (users)
(WHERE
(and
(or
(eq "login" login)
(eq "email" email)
)
(eq "password" (bcrypt password))
)
)
)
Has anyone here used the PostgreSQL package for temporal database support?
Would be interested in thoughts :)
http://temporal.projects.pgfoundry.org/reference.html
http://temporal.projects.pgfoundry.org/reference.html
Video quality sucks hard.
Is the source code from presentation available anywhere?
InfoQ is a pain in the ass, but if you open an account with their crappy site, as mentioned by others, you can download the presentation slides at the least. Not sure if the code mentioned will be clear, or even copy-pasteable. I know this is probably not what you are looking for, but that is probably the best you can do.
The code in the video is impossible to read. Are there higher-quality versions of this video anywhere?
FAT -> GIT
SQL -> Datomic?
SQL -> Datomic?