Reverse-engineering the Kayak app with mitmproxy(shubhro.com)
shubhro.com
Reverse-engineering the Kayak app with mitmproxy
http://www.shubhro.com/2014/12/18/reverse-engineering-kayak-mitmproxy/
10 comments
I've been MITM'ing my connections ever since I discovered the Proxomitron ( http://en.wikipedia.org/wiki/Proxomitron ) - it lets me customise webpages and filter out the crap before it gets to any browser, and works in all the browsers on the system; even the ones embedded in apps. It has a logging feature to show traffic too.
It would be great if a proxy could be run directly on the mobile device, as then it could be used anywhere along with the apps its monitoring.
As you said HTTPS is a bit of a pain since it's actively designed to resist such "attacks", but as long as there's the option to specify your own CAs it will work. Certificate "unpinning" is still a manual process, however...
It would be great if a proxy could be run directly on the mobile device, as then it could be used anywhere along with the apps its monitoring.
As you said HTTPS is a bit of a pain since it's actively designed to resist such "attacks", but as long as there's the option to specify your own CAs it will work. Certificate "unpinning" is still a manual process, however...
How would you use a proxy on a mobile phone? Just curious. I'm not familiar with Proxomitron, so I assume adding filters like privoxy or captures?
People have done this in ways by hacking the built-in VPN system I believe.
People have done this in ways by hacking the built-in VPN system I believe.
You just point to your Proxy server under Settings / WiFi. Alternatively you may want to install your proxy SSL CA on your mobile device, so you can see inside HTTPS connections (if supported).
Charles Proxy has some easy-to-use pointers [1][2].
[1] http://www.charlesproxy.com/documentation/faqs/ssl-connectio...
[2] http://www.charlesproxy.com/documentation/faqs/using-charles...
Charles Proxy has some easy-to-use pointers [1][2].
[1] http://www.charlesproxy.com/documentation/faqs/ssl-connectio...
[2] http://www.charlesproxy.com/documentation/faqs/using-charles...
Is Proxomitron any better than Privoxy?
Could you give a specific example of how socat can be used with the openssl binary in the way you suggest? I'd like to try it.
Personally I would just use the Firefox web developer tools rather than mitmproxy, much easier.
To log app requests? Firefox doesn't have a proxy unless I'm mistaken.
There are at least three addons that allow viewing network activity with Firefox:
https://getfirebug.com/ http://chrispederick.com/work/web-developer/ https://addons.mozilla.org/en-US/firefox/addon/live-http-hea...
I haven't tried it but I expect the developer edition does too:
https://www.mozilla.org/firefox/developer/
https://getfirebug.com/ http://chrispederick.com/work/web-developer/ https://addons.mozilla.org/en-US/firefox/addon/live-http-hea...
I haven't tried it but I expect the developer edition does too:
https://www.mozilla.org/firefox/developer/
But is there any way to use those to inspect traffic originating from outside of Firefox? With mitmproxy (or other tools, such as Charles/Fiddler/etc) you can run pretty much anything through them.
I would just use wireshark for that.
And so we're back to the original article. How do you deal with encrypted connections?
Another great mitm proxy is Charles Proxy http://www.charlesproxy.com/.
It has a really nice UI for looking at JSON responses such as the Kayak. Sometimes a collapsible tree is invaluable in looking through a response.
The easy filtering and formatting is primarily why I like it so much. Here is how it handles SSL for various ways http://www.charlesproxy.com/documentation/using-charles/ssl-...
Here is a screen shot of my iPhone Kayak app request for comparison http://imgur.com/gvKB6fr
It has a really nice UI for looking at JSON responses such as the Kayak. Sometimes a collapsible tree is invaluable in looking through a response.
The easy filtering and formatting is primarily why I like it so much. Here is how it handles SSL for various ways http://www.charlesproxy.com/documentation/using-charles/ssl-...
Here is a screen shot of my iPhone Kayak app request for comparison http://imgur.com/gvKB6fr
How is it better than mitmproxy? Also, it's not free/open-source.
In this case you get what you pay for.
You get filtering. Local and remote mapping. SSL decryption. Request and response detail views. Breakpoints. The list goes on. Seriously, it's the best $50 I spent as part of my developer toolkit.
I love Charles Proxy, I use it all the time for debugging service issues. Breakpoints and the ability to easily mock responses is really invaluable.
I don't know about Kayak's economics, but at least at Hipmunk we pay our data providers per search and it's really quite expensive. If they aren't offering an API anymore, it's probably because it was too pricey to operate.
You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.
You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.
I'm happy to pay to use a flight search API. Kayak and Hipmunk could provide an API and pass on the cost to the API consumers.
Sabre has a free flight search API: https://developer.sabre.com/docs/read/REST_APIs
The problem with any paid API is that the API consumer needs to make money somehow so that he can offer the price of the service.
How will the API consumer make money? I don't think Ads will generate enough revenue -- especially not on mobile.
Kayak will get some revenue from the bookings resulting from the searches, and would need to circle back some of this revenue back to the API consumer. This is technically complicated because it requires a lot of tracking infrastructure.
In the end, a service that would make sense is a lot more complicated than simply having an endpoint and saying it's X dollars per Y calls.
How will the API consumer make money? I don't think Ads will generate enough revenue -- especially not on mobile.
Kayak will get some revenue from the bookings resulting from the searches, and would need to circle back some of this revenue back to the API consumer. This is technically complicated because it requires a lot of tracking infrastructure.
In the end, a service that would make sense is a lot more complicated than simply having an endpoint and saying it's X dollars per Y calls.
I don't need to make money. Access to a flight API would be useful for planning my own trips.
[deleted]
How is using the API any different to using the website or app? Sounds like you need to move your access controls one layer down, into the API. If the app does rate limiting, the API should too/instead.
Take the github approach and let open source devs write innovative apps based on your open API!
Take the github approach and let open source devs write innovative apps based on your open API!
The difference is that Kayak is not able to show you ads if you use the API, so they generate less income from API usage compared to (web)app usage.
> From that folder, get the mitmproxy-ca-cert.pem file onto your mobile device by emailing it to yourself, for example. Then follow certificate installation steps for iOS or Android.
You can just go to http://mitm.it on the device. It's a 'magic domain' for the proxied host. See http://mitm.it/doc/certinstall/webapp.html
You can just go to http://mitm.it on the device. It's a 'magic domain' for the proxied host. See http://mitm.it/doc/certinstall/webapp.html
I've tried this website on iPhone 6, and it redirects to the standard version of the website instead.
Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.
There are two obvious issues, as I see it. #1 is whether someone's liable for poking around inside the Kayak app in the way the author of the post did. #2 is whether it's legal to "open-source" the API by disclosing this information, once obtained.
My quick take is that #1 likely violates Kayak's terms of use for the app, its web site, or both. The TOU for the website has a host of prohibitions, including don't "bypass or circumvent other measures employed to prevent or limit access to Our Website," no deep-linking (such as a link to the API endpoint), and as a catchall also prohibits all inappropriate-as-defined-by-Kayak activity. I haven't read the app TOU, but it would be standard practice to prohibit reverse engineering of the app itself.
So we don't really need to reach point #2, which raises some 1A issues the first doesn't. (Note I'm excluding DMCA and CFAA issues because Kayak isn't our sue-happy friends at the MPAA or RIAA.)
As a practical matter, though, the author of the linked post, Shubhro Saha, appears to be an undergrad, so is probably judgment-proof and not the most likely target of litigation.
My quick take is that #1 likely violates Kayak's terms of use for the app, its web site, or both. The TOU for the website has a host of prohibitions, including don't "bypass or circumvent other measures employed to prevent or limit access to Our Website," no deep-linking (such as a link to the API endpoint), and as a catchall also prohibits all inappropriate-as-defined-by-Kayak activity. I haven't read the app TOU, but it would be standard practice to prohibit reverse engineering of the app itself.
So we don't really need to reach point #2, which raises some 1A issues the first doesn't. (Note I'm excluding DMCA and CFAA issues because Kayak isn't our sue-happy friends at the MPAA or RIAA.)
As a practical matter, though, the author of the linked post, Shubhro Saha, appears to be an undergrad, so is probably judgment-proof and not the most likely target of litigation.
Just to add to this: the relevant case law seems to be Bowers v. Baystate Technologies [1], which held that EULA clauses prohibiting reverse engineering are enforceable. I personally think this is a terrible decision, since it also prevents a lot of good uses of reverse engineering (see some examples, like fighting censorship and diagnosing software vulnerabilities, in a recent paper of mine [2]).
[1] http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies [2] https://mice.cs.columbia.edu/getTechreport.php?techreportID=...
[1] http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies [2] https://mice.cs.columbia.edu/getTechreport.php?techreportID=...
DMCA has an exception for "interoperability" to allow reverse-engineering, which AFAIK is meant specifically for this case.
However, I wouldn't consider this any real "reverse-engineering", but just inspecting network traffic --- and it should be absolutely legal to do so when the traffic is generated from an app you legally obtained, running on a device you own.
IANAL, etc.
However, I wouldn't consider this any real "reverse-engineering", but just inspecting network traffic --- and it should be absolutely legal to do so when the traffic is generated from an app you legally obtained, running on a device you own.
IANAL, etc.
I'm not sure DMCA applies here, as it only bans tools that circumvent technical measures intended to enforce copyright protections. I don't think that Kayak's data is copyrightable (on the basis of Feist Publications, Inc., v. Rural Telephone Service Co., which says that mere compilations of public information with no creative aspect aren't considered copyrightable), so DMCA shouldn't apply.
But likewise, IANAL.
But likewise, IANAL.
I'm not a legal expert, but I bet there is a clause in TOS to prevent that. I wonder if anyone actually got sued for violating TOS this way though.
Great writeup, there's definitely potential to do something cool, especially if it's possible to get around any tracking and the following price manipulation. I tried installing your demo client, however on running it I got the following error:
root@kayak:~/kayak-mobile-client# python client.py
Departure airport code: LBG
Destination airport code: HAM
Departure date (MM/DD/YY): 12/26/14
Traceback (most recent call last):
File "client.py", line 56, in <module>
searchid = json.loads(r.text)["searchid"]
KeyError: 'searchid'Appreciate your kind words.
Have you set up your UUID and HASH according to the post? You'll actually need to run the app through mitmproxy yourself to get your unique credentials. Then, just drop those into the variables at the top of the script.
I just added some error handling to the script to make this a bit more obvious!
Have you set up your UUID and HASH according to the post? You'll actually need to run the app through mitmproxy yourself to get your unique credentials. Then, just drop those into the variables at the top of the script.
I just added some error handling to the script to make this a bit more obvious!
A useful technique for sure. The only technique I know to slow this down is to use certificate pinning, but it's probably pointless. Some of your users are probably extremely motivated (like ours [1]) and it's obvious to them that what they are doing is unsupported.
1. http://difm.eu/dox/
1. http://difm.eu/dox/
Certificate pinning would help to prevent sniffing traffic with tools like mitmproxy, but once you decompile the binary, you can disable certificate validation.
I'm curious, in your case did you consider preventing using your API by third-party clients?
I'm curious, in your case did you consider preventing using your API by third-party clients?
Interesting! We've used a similar technique to reverse-engineering mobile apps from different banks in order to create a Ruby gem to fetch bank data (balance and transactions).
https://github.com/ismaGNU/bankScrap
https://github.com/ismaGNU/bankScrap
I wouldn't call this reverse-engineering.
How are UUID and HASH are generated? Are they unique to every installation?
How are UUID and HASH are generated? Are they unique to every installation?
The same results can be achieved using only socat and the openssl binary.
While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic.
I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary.
I wish more users would view their own traffic.
Keep up the good work.