An alternative to fat JARs(product.hubspot.com)
product.hubspot.com
An alternative to fat JARs
http://product.hubspot.com/blog/the-fault-in-our-jars-why-we-stopped-building-uber-jars
15 comments
This similar to our setup with the slimfast-plugin. We use the maven-jar-plugin to add the Main-Class and Class-Path entries to the manifest. At build time, we use the upload goal of the slimfast-plugin to upload the dependencies. It automatically reads the configuration of the maven-jar-plugin to make sure the paths it generates match the classpath in the manifest. Then it spits out a JSON file that has info on each dependency, including its location in S3, file size, checksum, and relative path where it needs to be copied to match the Class-Path in the manifest (the JSON entries look like this https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e...).
On deploy, the simplest way to get up and running is to use the download goal of the slimfast-plugin. It reads this JSON file, downloads each dependency (using an optional cache folder), verifies the file size and checksum, and copies it to the correct relative path. The application will then start up happily with java -jar.
At HubSpot, we instead integrated this download step more transparently into our deploy process. At build time we read this JSON file and store the dependency information in our build database. Our deploy infrastructure already accepts S3 URLs and handles downloading, caching, verifying checksums, and copying to arbitrary directories so we just piggybacked on this existing functionality to have it download the application plus all of its dependencies for us.
On deploy, the simplest way to get up and running is to use the download goal of the slimfast-plugin. It reads this JSON file, downloads each dependency (using an optional cache folder), verifies the file size and checksum, and copies it to the correct relative path. The application will then start up happily with java -jar.
At HubSpot, we instead integrated this download step more transparently into our deploy process. At build time we read this JSON file and store the dependency information in our build database. Our deploy infrastructure already accepts S3 URLs and handles downloading, caching, verifying checksums, and copying to arbitrary directories so we just piggybacked on this existing functionality to have it download the application plus all of its dependencies for us.
Here's my very rough attempt in Gradle:
https://bitbucket.org/twic/ensure
Turned out to be more than 20 lines, but then i did it in Java rather than Groovy.
I do the downloads from the Maven repo (which should be your internal Maven repo!), so there's no need for an upload step. Deployment is done with a shell script with a few undemanding dependencies - unzip, curl, and openssl to verify the digests. I use a TSV file rather than JSON because it's easier for the shell script to read!
My next trick will be to figure out how to get this into a Cloud Foundry buildpack ...
Oh, the name: https://ensure.com/nutrition-products
https://bitbucket.org/twic/ensure
Turned out to be more than 20 lines, but then i did it in Java rather than Groovy.
I do the downloads from the Maven repo (which should be your internal Maven repo!), so there's no need for an upload step. Deployment is done with a shell script with a few undemanding dependencies - unzip, curl, and openssl to verify the digests. I use a TSV file rather than JSON because it's easier for the shell script to read!
My next trick will be to figure out how to get this into a Cloud Foundry buildpack ...
Oh, the name: https://ensure.com/nutrition-products
Here is something I mentioned on the reddit post:
You can go just as far with http://www.capsule.io/ and the capulet to import dependencies with Maven (from a Nexus proxy that also fronts S3)
You are now deploying thin Jars and having the artifacts cached on the local host through maven~!
You can go just as far with http://www.capsule.io/ and the capulet to import dependencies with Maven (from a Nexus proxy that also fronts S3)
You are now deploying thin Jars and having the artifacts cached on the local host through maven~!
Sounds like an interesting idea. I would be even more excited to see someone leverage Gradle for this. Assuming you publish your custom jar artifacts from your build server, your "deployment artifact" could be a single Gradle file that acts a your JAR dependency definition and execution wrapper. For deployment, sync the Gradle file and execute the wrapper to launch the service with Gradle handling all dependency updates and caching.
It doesn't use S3, but you can do something similar with Coursier:
https://github.com/alexarchambault/coursier#launch
https://github.com/alexarchambault/coursier#launch
Cool! We've been doing something similar with public Maven repositories as the backing store.
Jonathan: if you're listening, have you guys looked at Maven as a source instead of an S3 upload? What percentage of your dependencies are internal vs. external? We get 10-20x space savings from public-Maven-only elision.
EDIT: we've been working on a similar technique for executable JARs as well, which shares the same general optimization numbers and approach. It's a bit trickier, since we want to preserve the same 'java -jar' syntax, so requires a bit of classloader tomfoolery, and corresponding classloader awareness in code.
Jonathan: if you're listening, have you guys looked at Maven as a source instead of an S3 upload? What percentage of your dependencies are internal vs. external? We get 10-20x space savings from public-Maven-only elision.
EDIT: we've been working on a similar technique for executable JARs as well, which shares the same general optimization numbers and approach. It's a bit trickier, since we want to preserve the same 'java -jar' syntax, so requires a bit of classloader tomfoolery, and corresponding classloader awareness in code.
We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc.
We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.
We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.
I would recommend against depending on public repositories (of any form) as a backing store for repeatable builds. We've repeatedly seen public repositories remove packages, disappear entirely or otherwise make third party projects a release depends on unavailable. If bringing up a production service requires the presence of Maven, it will suffer an outage.
Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.
Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.
You can run Nexus or Archiva or some other Maven artifact mirror within your network to get the best of both worlds -- maven builds and none of your stuff will break if central is down.
Since those are proxy caches, they don't provide repeatable builds. The referred to release can be changed without your knowledge and Nexus will happily serve that to you as correct.
I don't follow. I shouldn't have mentioned Nexus as I'm a lot more familiar with Archiva. But,
- if the release of the same version was changed upstream, and you want this change to be picked up, that's not a repeatable build anyway.
- if the release was changed upstream and you don't want to pick it up, configure Archiva to fail on checksum mismatch [1].
- or as an alternative to enforcing the checksum, configure your local repo as internal instead of proxied, and handle it all yourself [2].
[1] https://archiva.apache.org/docs/2.2.1/adminguide/proxy-conne...
[2] https://archiva.apache.org/docs/2.2.1/adminguide/repositorie...
- if the release of the same version was changed upstream, and you want this change to be picked up, that's not a repeatable build anyway.
- if the release was changed upstream and you don't want to pick it up, configure Archiva to fail on checksum mismatch [1].
- or as an alternative to enforcing the checksum, configure your local repo as internal instead of proxied, and handle it all yourself [2].
[1] https://archiva.apache.org/docs/2.2.1/adminguide/proxy-conne...
[2] https://archiva.apache.org/docs/2.2.1/adminguide/repositorie...
If Nexus doesn't have that functionality, you can probably put an Nginx reverse proxy behind it that will always serve local files if available.
We maintain and check SHAs of the dependencies for that reason.
The iffy thing here (if I understand you correctly) is creating a dependency in your deployments that requires the maven servers to be alive, which isn't an uncommon kind of thing, but just saying - larger companies usually run their own package manager mirrors.
Capsule.io supports Maven as a source, and requires no classloader tomfoolery. It also supports native libraries, either embedded or in a Maven repo, java and native agents, and automatically applies JVM flags embedded in the manifest. It's got Maven, Gradle and Leiningen plugins to assist in capsule creation.
I really like your approach, is there any code you can share on that?
Getting it open-sourced has been a bit of a back-burner thing; I'll see if I can stoke the fires a bit.
If it is not open sourced yet, can you at least elaborate a little more on the inner workings?
Going from memory, we do something like this:
For WARs: In a mvn build step:
- identify candidate dependencies to exclude from the WAR. This includes a filter step to keep internal dependencies in the WEB-INF dir.
- generate a JSON manifest of all the excluded dependencies, including their local SHAs, and store the manifest in WEB-INF.
- copy a shell script into the new Maven Archive (MAR) that knows how to reverse the process above and output a WAR
Then, in our build process, we pass around the MAR files until the moment of deploy, at which time we expand back into a WAR and deploy.
We looked at just deploying the MARs themselves, but the way our containers manage file cache space, it was easier to rehydrate them pre-deploy.
For executable JARs: again, we start in a Maven build phase, and again identify the dependencies to include vs. the dependencies to exclude and fetch later. We create a totally-new JAR that contains all the classes in the thin Maven-built jar (I believe as a referenced JAR, not as expanded classes). We then bundle all the included JARs directly into the target JAR, and write a CSV manifest identifying which external dependencies are needed at runtime. We then copy a raw .class file with no dependencies [1] into the target JAR and write a manifest that identifies that class as the Main-Class. That class is responsible for invoking 'mvn' via shell execution to load dependencies, building a classloader with the right classpath, etc., and then handing off to the original Main-Class.
[1] hence CSV instead of JSON
For WARs: In a mvn build step:
- identify candidate dependencies to exclude from the WAR. This includes a filter step to keep internal dependencies in the WEB-INF dir.
- generate a JSON manifest of all the excluded dependencies, including their local SHAs, and store the manifest in WEB-INF.
- copy a shell script into the new Maven Archive (MAR) that knows how to reverse the process above and output a WAR
Then, in our build process, we pass around the MAR files until the moment of deploy, at which time we expand back into a WAR and deploy.
We looked at just deploying the MARs themselves, but the way our containers manage file cache space, it was easier to rehydrate them pre-deploy.
For executable JARs: again, we start in a Maven build phase, and again identify the dependencies to include vs. the dependencies to exclude and fetch later. We create a totally-new JAR that contains all the classes in the thin Maven-built jar (I believe as a referenced JAR, not as expanded classes). We then bundle all the included JARs directly into the target JAR, and write a CSV manifest identifying which external dependencies are needed at runtime. We then copy a raw .class file with no dependencies [1] into the target JAR and write a manifest that identifies that class as the Main-Class. That class is responsible for invoking 'mvn' via shell execution to load dependencies, building a classloader with the right classpath, etc., and then handing off to the original Main-Class.
[1] hence CSV instead of JSON
It's neat. But is it worth it? I thought inbound for S3 was free so the 150 Mb doesn't cost much. You're paying $0.25 a month to store up to a gig of fat jars so really nkthing. It's 20-ish seconds faster. Assuming a CI build, that doesn't matter. It is a more complex solution. This makes it more brittle. Which makes it more bad. Finally you're losing the single deployable. When something goes bad, you can't just download the jar. You have to fish through the maven dependencies.
> It's neat. But is it worth it?
Saving traffic is good in general, insofar as it speeds deploys and allows higher container density.
But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect.
I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it.
Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.
Saving traffic is good in general, insofar as it speeds deploys and allows higher container density.
But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect.
I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it.
Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.
Agreed, we might not have done it if we had to resort to having the application download its own dependencies at startup. In our case, our Mesos scheduler, Singularity, handles S3 artifacts at deploy time for us. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. By the time the app starts up, all of its dependencies are guaranteed to be present (if an S3 download failed, the deploy would have failed) so it's totally transparent to the application.
That makes sense.
I'm used to having to worry about disconnected environments (I worked on Cloud Foundry buildpacks for a while), for which Spring Boot's JARs work a treat.
In a fully connected environment this approach looks promising. If I bump into any of the Spring folks I will mention it.
I'm used to having to worry about disconnected environments (I worked on Cloud Foundry buildpacks for a while), for which Spring Boot's JARs work a treat.
In a fully connected environment this approach looks promising. If I bump into any of the Spring folks I will mention it.
20-30 seconds ends up being 50% for most of our Java builds, so it was a huge speedup. It also saves 20-30 seconds at deploy time when we need to download this file again. And with lots of concurrent deploys we were occasionally seeing these large downloads saturate the NIC on our application servers.
Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.
When something goes wrong we rarely downloaded the JAR, but if you really wanted to you could download the thin JAR and use the SlimFast metadata bundled with it to download the exact dependencies it would get when deployed.
Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.
When something goes wrong we rarely downloaded the JAR, but if you really wanted to you could download the thin JAR and use the SlimFast metadata bundled with it to download the exact dependencies it would get when deployed.
I will respectfully disagree and say it's worth it in this case.
Here's the reasons I see that makes it worth it:
* It reduces the build time to less than 50%
* They are doing this for 1000+ java applications
* The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.
Here's the reasons I see that makes it worth it:
* It reduces the build time to less than 50%
* They are doing this for 1000+ java applications
* The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.
So the real question is.. do they NEED 1000-2000 java applications?
Or could it be much more sanely organized with say 15 or 20 applications?
I know nothing about the codebase but it sounds like they may have nanoservices...
Or could it be much more sanely organized with say 15 or 20 applications?
I know nothing about the codebase but it sounds like they may have nanoservices...
It's roughly 300 REST APIs and the rest are cron jobs, kafka workers, Hadoop or Spark jobs, etc. etc. Each part of the product gets its own API which is where most of them come from as our product is quite broad
It's hard for me to understand why in 2016 200MB files are still any kind of real problem.
From the article:
"Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."
"Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."
I wasn't being critical of the OPs build process. I agree that uploading 200 MB to S3 is slow, but it shouldn't be.
Back in the day , used to have all common dependency jars in server/lib folder and only deploy slim application war to the server . Reminds me of this
What was the process for adding or removing a dependency? Changing a version? If two apps needed different versions?
You added the dependency to server/lib. You standardized on versions, and applied only security fixes, testing them in a separate environment first. You only upgraded versions of libs if you had a critical feature you couldn't live without, of if your existing libs were about to be deprecated and out-of-support.
I'm not saying it wasn't a dark time, but it had its merits and drawbacks just like everything else.
I'm not saying it wasn't a dark time, but it had its merits and drawbacks just like everything else.
and remember how bad it was trying to get application server X to deploy your war file from a command line without exploding for some bizarre reason? It still actually makes me sad to think of the hours of my life wasted writing WLST jython scripts trying to get WebLogic to behave properly.
Reinventing OSGi configurations.
We're using ServiceMix as a container for services (micro or otherwise) and while OSGI isn't without its own annoyances, it really is nice once you get everything working.
The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. Recently I specifically had an issue with Spring in this regard (yeah, yeah, don't use Spring, I know... but there was a specific reason in that case, at least initially) that would have been easy to resolve if the Spring guys still distributed OSGI'ified versions of their jars.
Still, all in all, I'm pretty happy with this approach.
The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. Recently I specifically had an issue with Spring in this regard (yeah, yeah, don't use Spring, I know... but there was a specific reason in that case, at least initially) that would have been easy to resolve if the Spring guys still distributed OSGI'ified versions of their jars.
Still, all in all, I'm pretty happy with this approach.
> The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information.
It's a mess, and I fear Java 9/10's modularization approach will even make it worse (so that instead of P2 (Eclipse's Equinox' OSGi bundles) and Maven, we'll have another set ... sigh. I'm still looking for a solution that transparently translates Maven metainformation to OSGi bundle information at some higher level (for example by hooking into the OSGi container's dependency resolution logic).
Edit: apparently, such a mechanism exists (ResolverHook)
It's a mess, and I fear Java 9/10's modularization approach will even make it worse (so that instead of P2 (Eclipse's Equinox' OSGi bundles) and Maven, we'll have another set ... sigh. I'm still looking for a solution that transparently translates Maven metainformation to OSGi bundle information at some higher level (for example by hooking into the OSGi container's dependency resolution logic).
Edit: apparently, such a mechanism exists (ResolverHook)
isn't that exactly what wrap is for?
That has mostly worked well for me when dealing with non-OSGI jars. Once I figured out how to set things up using the features.xml file and specify wrap: for the generic jars, things worked nicely.
All of this said, I'm fairly new to OSGI, so I may still run into some dark corners that will turn me off. But right now, it seems to be serving the purpose well.
All of this said, I'm fairly new to OSGI, so I may still run into some dark corners that will turn me off. But right now, it seems to be serving the purpose well.
I've been working with Fuse for ages and I'm still not convinced OSGi is worth the trouble. ServiceMix, Felix, Karaf etc can be a total lottery some times as to whether or not it will do what you asked. Dropwizard or SpringBoot ftw!
First thought (to the article title "Why we stopped building fatjars") was that the build person had "graduated" and took their newly "acquired superpowers elsewhere" [0]
Joking aside, fantastic idea =)
Have you received a cease and desist from SlimFast yet for violating their trademark?
[0] - http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyo...
Joking aside, fantastic idea =)
Have you received a cease and desist from SlimFast yet for violating their trademark?
[0] - http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyo...
Don't believe everything you read -- especially when there's a single source.
I did my first fat jar project in 2010 with embedded jersey and it was about 10mb. We use dropwizard now and most of our microservices sit at around 20mb which I think is still slim enough to avoid the extra overhead of managing the dependencies on deployment. IMHO Its hard to overstate how much benefit you get from java -jar
You can have both. Capsule.io (I'm a developer) gives you the option of either embedding all dependencies in the JAR (including native libraries, if you need them), or just listing Maven dependencies and having them downloaded, cached, and shared on the first launch. It also lets you supply all the JVM flags directly in the JAR manifest.
I'll have to check it out, thanks!
We still use java -jar with this setup. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR with no runtime or ClassLoader funny business.
Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.
Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.
> We have over 1,000 microservices constantly being built and deployed.
Wait until they find out how much faster in-process function calls are than RPCs.
Wait until they find out how much faster in-process function calls are than RPCs.
Nobody said that calls between microservices have to be done via RPCs.
I use a framework at Google that is microservice-based, and allows for composition of microservices into an assembly. Within an assembly, calls between services are done in-process; if you are calling a service in a different assembly, it has to make an RPC.
The mechanics of whether it's a local call or an RPC are all handled at the framework leve. This makes splitting a service into its own assembly a simple task (don't have to update any code, just configuration; this can be managed by SREs).
I use a framework at Google that is microservice-based, and allows for composition of microservices into an assembly. Within an assembly, calls between services are done in-process; if you are calling a service in a different assembly, it has to make an RPC.
The mechanics of whether it's a local call or an RPC are all handled at the framework leve. This makes splitting a service into its own assembly a simple task (don't have to update any code, just configuration; this can be managed by SREs).
>I use a framework at Google that is microservice-based, and allows for composition of microservices into an assembly. Within an assembly, calls between services are done in-process; if you are calling a service in a different assembly, it has to make an RPC.
Sounds a lot like OTP.
In Java-land, you could also just have your client and service use the same interface. Just inject different implementations for local and remote.
Sounds a lot like OTP.
In Java-land, you could also just have your client and service use the same interface. Just inject different implementations for local and remote.
Right, then you can have some supporting code automatically handle the remoting.
Once you've gone to the trouble of automatically generating the glue between the local interface and the remote object, it becomes simple to interpose further useful things - security checks, transactions if you're using a relational database, and so on. You can even have the glue code lazily instantiate the service components, so you don't need to actually load them until a client makes a call.
You'd want a cool name for it, though. Extreme Java Binding? Enhanced Jet Bus? Excellent Joy Bringer?
Once you've gone to the trouble of automatically generating the glue between the local interface and the remote object, it becomes simple to interpose further useful things - security checks, transactions if you're using a relational database, and so on. You can even have the glue code lazily instantiate the service components, so you don't need to actually load them until a client makes a call.
You'd want a cool name for it, though. Extreme Java Binding? Enhanced Jet Bus? Excellent Joy Bringer?
Wait until they realized microservices can be deployed independently and have strict APIs without people using classes they shouldn't or creating spaghetti code because they have 800k LOC.
> without people using classes they shouldn't or creating spaghetti code because they have 800k LOC.
Honestly, if you can't trust your devs not to make a mess in a monolith, you can't trust them not to make a distributed mess with microservices.
Both architectural patterns have their uses. Neither will save you from bad developers.
Honestly, if you can't trust your devs not to make a mess in a monolith, you can't trust them not to make a distributed mess with microservices.
Both architectural patterns have their uses. Neither will save you from bad developers.
I think monoliths lend themselves to hacks more often than microservices do. It's too easy to just import something you shouldn't, where as breaking the rest API is a lot harder. You can't just do a bad database access because there's usually a network firewall that will keep you from doing your tomfoolery on the DB.
I also could be wrong about this, but I like the microservice after projects reach a certain size.
I also could be wrong about this, but I like the microservice after projects reach a certain size.
I refer to this as "design backpressure". In a microservice world you need to consciously decide to cross the network boundary.
With a single memory space it's easy to simply go around whatever architecture you "should" be following.
Microservices don't solve balls of mud, but they can make it harder to form and easier to spot forming.
With a single memory space it's easy to simply go around whatever architecture you "should" be following.
Microservices don't solve balls of mud, but they can make it harder to form and easier to spot forming.
The problem with API is that is not an easy to create a good one. It's so easy to do it wrong, and once you do, it's very hard to change it later. API should be created pretty much by a committee where everyone from the company can have an input how it will affect them.
It's not a panacea, it looks like you're just pushing the complexity into another place.
It's not a panacea, it looks like you're just pushing the complexity into another place.
At least with an API, people seem to care more about consensus and seem to agree that it should be planned out. "Just make it work" seems to happen to the guts.
And no, not panacea, just a way of thinking that I think is useful.
And no, not panacea, just a way of thinking that I think is useful.
"Microservices" often lead to the degenerate fragmentary case (look at people bragging about how many microservices they have--that's a weird mindset to have).
I am a big fan of a well-designed SOA, but that's not properly bloggable here in 2016.
I am a big fan of a well-designed SOA, but that's not properly bloggable here in 2016.
That's what module systems and access modifiers are for.
Now, OK, the JVM ecosystem lacks a good module system at the moment. But Java 9 will fix that. There is also OSGi though I don't know much about it.
Now, OK, the JVM ecosystem lacks a good module system at the moment. But Java 9 will fix that. There is also OSGi though I don't know much about it.
Wait until they realized that solving their spaghetti problem with microservices resulted in them operating an Italian restaurant.
Pushing complexity onto interprocess/machine communication will just result in higher levels of spaghetti. Do you really think that same people who would construct a monolith poorly would do a better job building a distributed system instead?
I don't mean to be too dogmatic; there are legit use cases for services. I am just highly skeptical of microservices as a solution for complexity.
Pushing complexity onto interprocess/machine communication will just result in higher levels of spaghetti. Do you really think that same people who would construct a monolith poorly would do a better job building a distributed system instead?
I don't mean to be too dogmatic; there are legit use cases for services. I am just highly skeptical of microservices as a solution for complexity.
I'm not dogmatic about either. 1000 microservices sounds... excessive. I was countering the point that optimizing for speed might lead to less maintainable infrastructure.
You don't need microservices for that. This is Java, we have working dependency management and multi-module projects via maven, and the system can actually enforce things like separation of interface and implementation.
> Wait until they find out how much faster in-process function calls are than RPCs.
2017 will be the year of the return of the monolith: "How We Replaced 946 Microservices With a Statically Linked 10kB Rust Binary and How It Ulimately Did Not Save Us From Our Complete Lack of Anything Resembling a Coherent Business Model".
2017 will be the year of the return of the monolith: "How We Replaced 946 Microservices With a Statically Linked 10kB Rust Binary and How It Ulimately Did Not Save Us From Our Complete Lack of Anything Resembling a Coherent Business Model".
Faster hardware is usually cheaper than hiring more developers.
That changes very quickly if you create distributed systems that tend to either have underdimensioned external IO (networks) and/or be excessively expensive. It's not unusual for systems to balloon up to several times the original size the moment you have to cross the boundary to communicating over the network. Even if it's a LAN.
>Faster hardware is usually cheaper than hiring more developers.
Now that Moore's law is running out of steam, we can reasonably expect that to become false within the next decade or so. The division between those who can merely write code and those who can properly engineer software systems will become all too apparent then.
Now that Moore's law is running out of steam, we can reasonably expect that to become false within the next decade or so. The division between those who can merely write code and those who can properly engineer software systems will become all too apparent then.
To deploy a process in my current project, I use Apache Ivy to resolve its dependencies (starting from a small set of top-level modules). Then I cache the results because Ivy resolution tends to take a while when you have a lot of dependencies and a lot of repositories.
Then I send a message to a remote agent with the command to run (classpath, main class, jvm args, args, configuration, etc). The agent then downloads the jars that are not already present in its cache with a valid SHA-1 checksum then starts.
Works quite well.
Then I send a message to a remote agent with the command to run (classpath, main class, jvm args, args, configuration, etc). The agent then downloads the jars that are not already present in its cache with a valid SHA-1 checksum then starts.
Works quite well.
Everyone uses fat JARs because it removes a lot of things that can go wrong operationally (especially when standing up new servers).
The approach described in this article is the way Java was designed to work; but then everyone said you needed a "self-executing" service binary for portability. Enter the fat JAR, containers, orchestration layers, etc.
JVM application servers are certainly the way to go if you run a bunch of Java. But then you're committing to maintaining a bunch of dependencies on your application servers, so really you're pushing the tech debt down the line to your ops folks who now have to ensure dependencies are synced, maven servers running and updated properly, proxies punched for new external dependencies, etc.
The approach described in this article is the way Java was designed to work; but then everyone said you needed a "self-executing" service binary for portability. Enter the fat JAR, containers, orchestration layers, etc.
JVM application servers are certainly the way to go if you run a bunch of Java. But then you're committing to maintaining a bunch of dependencies on your application servers, so really you're pushing the tech debt down the line to your ops folks who now have to ensure dependencies are synced, maven servers running and updated properly, proxies punched for new external dependencies, etc.
On limited information, this sounds like a work around on a root problem of runaway dependencies.
Maybe something like proguard could reduce deployable jar size to only the classes used.
I assume these are daily dev deploys and not production deploys.
Maybe something like proguard could reduce deployable jar size to only the classes used.
I assume these are daily dev deploys and not production deploys.
We definitely have some dependency cruft that could be trimmed (lots of relocated copies of Guava due to incompatibilities, for example).
We do frequent production deploys (of individual services, there's no such thing as deploying our entire application). To give an idea, it's a little before 1pm here and across our team there have been 180 production deploys already today.
We do frequent production deploys (of individual services, there's no such thing as deploying our entire application). To give an idea, it's a little before 1pm here and across our team there have been 180 production deploys already today.
You might be able to use a standard embedded POM instead of a dependency list, which might reduce the work a bit, but that would then require doing transitive dependency resolution at deploy time, which is probably a bad idea.
Phase I is something like 10 - 20 lines of Gradle, tops. You could pack it up into a plugin easily enough. Phase II is a similar amount of shell script, maybe more.
For extra safety, add SHA3 hashes to the dependency list file, and check them when you download the dependencies.
I've described this as using a Maven repo, but that doesn't mean it has to hit Nexus or whatever; you can just put JARs in S3 in the right layout. A while ago, i was doing this by maintaining a repository locally, and just pushing the whole thing to a Bitbucket website:
https://bitbucket.org/twic/twic.bitbucket.org/src/14ac48d4c4...
You could do something similar, perhaps going from Nexus to S3.