JSON Patch, a proposed standard for synchronizing JSON structures(tools.ietf.org)
tools.ietf.org
JSON Patch, a proposed standard for synchronizing JSON structures
http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-03
10 comments
There are already some implementations:
- https://github.com/dharmafly/jsonpatch.js
- https://github.com/stefankoegl/python-json-patch
This JavaScript implementation is up to date https://github.com/bruth/jsonpatch-js
(I am the author)
(I am the author)
Do you know of a good library for generating a patch from two JSON structures that, when applied to structure A, would produce structure B?
I implemented this a while back, but have not yet cleaned it up nor turned it into a library: http://jsfiddle.net/bruth/p6SUF/ Open up the console to see the output.
Here is slightly cleaner one that I plan to submit as a feature to Backbone. It assumes underscore.js is present: https://github.com/bruth/backbone/commit/a7f8bd3d9fa06a63cf4... Here is a JSFiddle example: http://jsfiddle.net/bruth/pwpNw/ (Open up the console to see the output)
EDIT: Add JSFiddle example for second implementation
Here is slightly cleaner one that I plan to submit as a feature to Backbone. It assumes underscore.js is present: https://github.com/bruth/backbone/commit/a7f8bd3d9fa06a63cf4... Here is a JSFiddle example: http://jsfiddle.net/bruth/pwpNw/ (Open up the console to see the output)
EDIT: Add JSFiddle example for second implementation
Related standard for pointing to values inside of a json file - http://tools.ietf.org/html/draft-ietf-appsawg-json-pointer-0...
Example:
Example:
http://example.com/example.json#/foo/inner%20object/baz
var temperature = GET("http://example.com/temperature.json#/europe/poland/warsaw);Oh nice. I came across JSON Pointer a couple of days ago (whilst working with JSON Schema). It didn't fully make sense at the time, but now you've put it like that, I can see how even that simple (GET-only) extension to your average REST API could be pretty handy for getting at commonly-used properties.
I may just implement it as a secret-squirrel feature in the API I'm working on right now.
I may just implement it as a secret-squirrel feature in the API I'm working on right now.
Are we going to get to a JSON as the media-type of choice - my big concern is that compared to XML, JSON does not support namespaces (hard to define your own media type (and let others parse it)) and gets really messy around the third level of indentation.
Don;t get me wrong, I prefer using it, but XML is better for those all important "here is my new type of doc"
Don;t get me wrong, I prefer using it, but XML is better for those all important "here is my new type of doc"
> "JSON ... gets really messy around the third level of indentation."
This is not a place where XML compares favorably.
This is not a place where XML compares favorably.
:-)
Messy as in the meaningfulness starts to deteriorate faster than with XML - the ability to have attributes and tags when you are deeply nested can be useful
However, if you are that deeply nested, maybe you are doing it wrong
Messy as in the meaningfulness starts to deteriorate faster than with XML - the ability to have attributes and tags when you are deeply nested can be useful
However, if you are that deeply nested, maybe you are doing it wrong
What is a real-world use case for this? While beneficial to size I do see a problem with bugs in implementation. A faulty patch apply procedure leads to incorrect patches when edited, which lead to invalid documents states etc... The benefit, I suppose, is the original document is left untouched. Is there an additional purpose to using patches?
One real-world use case is differential synchronization. Here's a Google Tech Talk on the subject:
https://www.youtube.com/watch?v=S2Hp_1jqpY8
This is for real-time collaborative editing or replication of structured data (unlike Operational Transformation, which is optimized for text). Sending patches back and forth is at the core of the protocol.
https://www.youtube.com/watch?v=S2Hp_1jqpY8
This is for real-time collaborative editing or replication of structured data (unlike Operational Transformation, which is optimized for text). Sending patches back and forth is at the core of the protocol.
You quickly run into the need for this if you build an API where clients can update individual fields within JSON documents instead of just replacing the whole JSON document with a PUT.
It can probably be argued that each individual field could also be represented by a unique URL, but IMO that makes things pretty complicated and you have to make a lot of unnecessary HTTP requests.
I still think it's a mistake to use / instead of . as the separator though. Using . is much more intuitive and it's already used in e.g. MongoDB queries.
It can probably be argued that each individual field could also be represented by a unique URL, but IMO that makes things pretty complicated and you have to make a lot of unnecessary HTTP requests.
I still think it's a mistake to use / instead of . as the separator though. Using . is much more intuitive and it's already used in e.g. MongoDB queries.
I implemented something like this to notify interested parties in realtime of changes to a shared data element for synchronization. It is basically eventual consistency
using op-based CRDTs.
Why slashes instead of dots? Wouldn't the key paths make much more sense as "foo.bar.bat".
Slashes are pretty common to describe paths (URLs, file systems, XML, etc.)
Yes, and with dots it would be less obvious that people are reinventing the whole XML ecosystem.
Someone should design a meta-language (probably based on s-expressions) that would easily represent all Object Notations and Markup Languages and an XPath/XSLT meta-equivalent to rule 'em all.
Someone should design a meta-language (probably based on s-expressions) that would easily represent all Object Notations and Markup Languages and an XPath/XSLT meta-equivalent to rule 'em all.
As always, there is a relevant XKCD:
http://xkcd.com/927/
http://xkcd.com/927/
I'm guessing that dots are marginally more likely to occur as keys, so they'd need to be escaped more often.
Why would you even need a json document to modify another json document ? Could you not author a DSL to reference XPATH like structures and have operations performed on them ?
Secondly I don't see what additional benefit this adds to JSON documents across various use cases (HTTP servers / parsers / persistence / etc). From an API's perspective, are parsers expected to accept this patch-document and apply them to json-documents that they already loaded ? The closest match you get to something like this in the XML world is a XSLT. Feed one XML document in and it can be converted to an XML or output of another kind. You cannot expect HTTP servers to accept these via PATCH and update documents atomically. Resource modification is not atomic in eventually consistent systems.
For those systems where an update / patch can be made atomically, updating a field on a document without replacing it makes sense. However doing it via another json document is not the way to go IMHO. Use a DSL instead.
Secondly I don't see what additional benefit this adds to JSON documents across various use cases (HTTP servers / parsers / persistence / etc). From an API's perspective, are parsers expected to accept this patch-document and apply them to json-documents that they already loaded ? The closest match you get to something like this in the XML world is a XSLT. Feed one XML document in and it can be converted to an XML or output of another kind. You cannot expect HTTP servers to accept these via PATCH and update documents atomically. Resource modification is not atomic in eventually consistent systems.
For those systems where an update / patch can be made atomically, updating a field on a document without replacing it makes sense. However doing it via another json document is not the way to go IMHO. Use a DSL instead.
So to be compatible with the PATCH method json-patch must be atomic. Barring some type of transactional memory system won't this mean that you either must a) copy the target before applying the patch, or b) record a reverse patch to roll-back with?
Seems there would only be benefit if the changes were very simple and the target very large.
Seems there would only be benefit if the changes were very simple and the target very large.
I only skimmed the intro, but why bring back the problem that REST was trying to solve? It may not be strictly RESTful, but partial updates to JSON resources is simpler... properties specified get updated, properties not specified remain as is, and setting a property undefined is akin to removal. Is that really flawed?
Hmm. I appreciate the idea of a standardized way to modify a JSON document, but I'm not certain what problem it realistically solves. Can someone who's worked with APIs that accept updates give a code example of something that would be better if you accepted patches instead?
Other commenters have brought up size - that the patch must be quite a bit smaller than the whole document in order to gain a benefit - but I'm not sure it's useful even then. JSON is typically the representation of an object for transmission over the wire, not its final representation on the server.
I have a Rails servers with a JSON API, but it stores information in a SQL database where the columns roughly correspond to the JSON keys. I'm not actually storing JSON in the database and the mapping of JSON key to database column is not always one-to-one. That means if I want to accept a patch for an object with JSON Patch, I need some way of applying the patch "upstream" to the SQL row that is the source of truth for the object. And, of course, for a database with a schema, only a strictly limited subset of potential JSON Patch objects submitted can actually be applied.
The architectural benefit of accepting JSON Patch as an update is that "it's just JSON", but in order to apply a patch on pure JSON, you must have pure JSON and store pure JSON. It's now common for servers to use JSON but that doesn't mean it's "JSON all the way down". In the case of my Rails project, I would need to generate the JSON for an object (or retrieve it from the cache), apply the patch to that JSON, then treat the result as if it had been submitted as an entire document to the update endpoint. Instead of parsing received JSON, I need to generate JSON, process a patch, then parse the result. Whether or not the client has to deal with the entire JSON document, the server has to do so.
Now moving the burden to the server can actually be a very important improvement if that's where you want to do the work, but I don't know that you need a new standard of JSON document to do that - if you want to perform small updates on large objects, you can write the server such that clients are allowed to omit key-value pairs that are not to be changed when sending JSON to update existing items. Perhaps I'm just dense, but it seems like you only need to iterate through the keys that are included and update only those fields.
JSON Patch is a defined format for doing that but it adds a layer of abstraction to the operations, when you actually would want clients to be concrete in specifying keys and values so the server doesn't have to perform lookup to figure out which value is under which path.
Likewise for the concept of "synchronizing" structures, the problem is that you cannot apply a patch correctly unless you start with the same JSON before applying the patch. I don't know if the speed gain from sending patches instead of entire documents justifies the extra reliability work you need to do to make sure the synchronization eventually succeeds. Plus, just as you don't save time in the code to apply a received patch unless a JSON document is the canonical "source of truth" for that object, it takes work to generate a patch for another party to apply unless a JSON document is the "source of truth" and you can cheaply produce the diff in terms of JSON. (Like any document format, you CAN process and produce the data eventually, but the question is whether it's easier than alternatives.)
It seems like this could be useful for a system where you specifically want to store the changes to an item - i.e. an audit log - but I don't know if JSON Patch is substantially better than existing ways of doing that. Before reading this paper, if I was asked to write a project that stores the changes to a chunk of JSON, I might have done something like insert a \n character after each key-value pair and store the history as commits in git. That's a hack, but since JSON is text and developers already have lots of ways to manipulate text, I'm not sure why it's necessarily better to work with the state changes as "JSON that is changed by events that are also JSON" rather than "text that is changed by events, and the result always parses as JSON", since there are already good tools for working with plain text and evaluating changes to the text.
I can see this being useful in a system where you have really set a deep architectural assumption that it's "JSON all the way down", and therefore a JSON Patch can be treated with some level of abstraction rather than just being a well-defined shorthand for sending fewer bytes over the wire. If the system is a document store that considers arbitrary JSON to be the unit of storage, then you don't need to worry about mapping into alternative storage formats. However, that seems like a very specific subset of the architectures in the world of "servers that communicate with JSON".
Other commenters have brought up size - that the patch must be quite a bit smaller than the whole document in order to gain a benefit - but I'm not sure it's useful even then. JSON is typically the representation of an object for transmission over the wire, not its final representation on the server.
I have a Rails servers with a JSON API, but it stores information in a SQL database where the columns roughly correspond to the JSON keys. I'm not actually storing JSON in the database and the mapping of JSON key to database column is not always one-to-one. That means if I want to accept a patch for an object with JSON Patch, I need some way of applying the patch "upstream" to the SQL row that is the source of truth for the object. And, of course, for a database with a schema, only a strictly limited subset of potential JSON Patch objects submitted can actually be applied.
The architectural benefit of accepting JSON Patch as an update is that "it's just JSON", but in order to apply a patch on pure JSON, you must have pure JSON and store pure JSON. It's now common for servers to use JSON but that doesn't mean it's "JSON all the way down". In the case of my Rails project, I would need to generate the JSON for an object (or retrieve it from the cache), apply the patch to that JSON, then treat the result as if it had been submitted as an entire document to the update endpoint. Instead of parsing received JSON, I need to generate JSON, process a patch, then parse the result. Whether or not the client has to deal with the entire JSON document, the server has to do so.
Now moving the burden to the server can actually be a very important improvement if that's where you want to do the work, but I don't know that you need a new standard of JSON document to do that - if you want to perform small updates on large objects, you can write the server such that clients are allowed to omit key-value pairs that are not to be changed when sending JSON to update existing items. Perhaps I'm just dense, but it seems like you only need to iterate through the keys that are included and update only those fields.
JSON Patch is a defined format for doing that but it adds a layer of abstraction to the operations, when you actually would want clients to be concrete in specifying keys and values so the server doesn't have to perform lookup to figure out which value is under which path.
Likewise for the concept of "synchronizing" structures, the problem is that you cannot apply a patch correctly unless you start with the same JSON before applying the patch. I don't know if the speed gain from sending patches instead of entire documents justifies the extra reliability work you need to do to make sure the synchronization eventually succeeds. Plus, just as you don't save time in the code to apply a received patch unless a JSON document is the canonical "source of truth" for that object, it takes work to generate a patch for another party to apply unless a JSON document is the "source of truth" and you can cheaply produce the diff in terms of JSON. (Like any document format, you CAN process and produce the data eventually, but the question is whether it's easier than alternatives.)
It seems like this could be useful for a system where you specifically want to store the changes to an item - i.e. an audit log - but I don't know if JSON Patch is substantially better than existing ways of doing that. Before reading this paper, if I was asked to write a project that stores the changes to a chunk of JSON, I might have done something like insert a \n character after each key-value pair and store the history as commits in git. That's a hack, but since JSON is text and developers already have lots of ways to manipulate text, I'm not sure why it's necessarily better to work with the state changes as "JSON that is changed by events that are also JSON" rather than "text that is changed by events, and the result always parses as JSON", since there are already good tools for working with plain text and evaluating changes to the text.
I can see this being useful in a system where you have really set a deep architectural assumption that it's "JSON all the way down", and therefore a JSON Patch can be treated with some level of abstraction rather than just being a well-defined shorthand for sending fewer bytes over the wire. If the system is a document store that considers arbitrary JSON to be the unit of storage, then you don't need to worry about mapping into alternative storage formats. However, that seems like a very specific subset of the architectures in the world of "servers that communicate with JSON".
> Can someone who's worked with APIs that accept updates give a code example of something that would be better if you accepted patches instead?
Concurrent edition of tabular data by several users.
User loads a HTML <form> with a bunch of widgets into her browser, edits some of the data and <submit>s. The usual implementation causes all data submitted via HTTP POST request, as simple KEY=VALUE dataset. Should two users send POSTs of same resource (same URI) concurrently, it is not clear what are real changes and what is unchanged, nor how to apply it to the resource.
What I'd like to do -- and indeed, some parts of my software does, albeit in non-standard way -- is submitting diff, in form of KEY=<original-value, new-value>. Makes it easier for the backend to serialize users' changes, and either apply them in atomic way or reject just the conflicting ones.
Concurrent edition of tabular data by several users.
User loads a HTML <form> with a bunch of widgets into her browser, edits some of the data and <submit>s. The usual implementation causes all data submitted via HTTP POST request, as simple KEY=VALUE dataset. Should two users send POSTs of same resource (same URI) concurrently, it is not clear what are real changes and what is unchanged, nor how to apply it to the resource.
What I'd like to do -- and indeed, some parts of my software does, albeit in non-standard way -- is submitting diff, in form of KEY=<original-value, new-value>. Makes it easier for the backend to serialize users' changes, and either apply them in atomic way or reject just the conflicting ones.
Thanks for the example. Receiving a patch does let you know just what change the user intended to make, and often you actually care about the change, not the before or after.
As I keep looking at this, it seems that it's most valuable when you consider the patch as an element of an event-based system, where you're storing or operating on changes rather than just the data. The reason it helps with a bulk editing process is that you want to know what changes Alice and Bob made if they both submit at the same time - operating on diffs simplified the system. For other systems where you operate on diffs, like an audit log, it's helpful to have a first-class format that defines the diff on a JSON document.
I think that's the biggest caveat I'm seeing with the proposal. It's pitched as a way to handle HTTP PATCH requests, but a format that defines "diffs on JSON documents" is most suited for systems that primarily store JSON documents and diffs on JSON documents - it is much less broadly applicable for "HTTP servers that accept JSON data", since frequently they only work with JSON over the wire, not as the ultimate source of truth. If you don't store JSON documents as the ultimate source of truth or have a domain-model reason to want diffs, I'm unsure of the advantage.
As I keep looking at this, it seems that it's most valuable when you consider the patch as an element of an event-based system, where you're storing or operating on changes rather than just the data. The reason it helps with a bulk editing process is that you want to know what changes Alice and Bob made if they both submit at the same time - operating on diffs simplified the system. For other systems where you operate on diffs, like an audit log, it's helpful to have a first-class format that defines the diff on a JSON document.
I think that's the biggest caveat I'm seeing with the proposal. It's pitched as a way to handle HTTP PATCH requests, but a format that defines "diffs on JSON documents" is most suited for systems that primarily store JSON documents and diffs on JSON documents - it is much less broadly applicable for "HTTP servers that accept JSON data", since frequently they only work with JSON over the wire, not as the ultimate source of truth. If you don't store JSON documents as the ultimate source of truth or have a domain-model reason to want diffs, I'm unsure of the advantage.
Agreed, I'm excited for this format, but only because I store almost all of my state in JSON blobs.
Reminds me of RDF's http://www.w3.org/DesignIssues/Diff