The full complaint against Oracle by the state of Oregon [pdf](doj.state.or.us)
doj.state.or.us
The full complaint against Oracle by the state of Oregon [pdf]
http://www.doj.state.or.us/releases/pdf/FINAL_Complaint_8_22_14.pdf
8 comments
In the requirements, OR seems pretty adamant that they want a COTS solution with no or very little custom code.
I'm sure they could do this project with OSS products and custom code integrating everything, but those don't come with support contracts. If there's no support contract that means OR has to support it itself, which means hiring people with deep knowledge of the system to quite possibly do nothing for long periods of time until a bug is found for their particular use case, because no one else is using the product.
That's not cost effective compared to an off the shelf solution from a vendor that can use 1 support staff to help multiple customers, especially when you realize this means requiring at least a security expert, a db expert, and a frontend UI/UX.
It's exactly the same reason why tech companies don't write everything themselves from OSS projects. Sure, they do that for certain things that they deem critical, but not for things like salesforce or the direct deposit and accounting software and sometimes not even for stuff as important and tech-ey as a vcs, like perforce.
I'll totally agree that OR should keep IT in house for this stuff, but I don't think it's a great idea for a state or business to suddenly get NIH syndrome and want to write custom software themselves for anything their agencies might need. Can you imagine if this had happened because OR did this project in house and had the same results? Or if the whitehouse had said they weren't outsourcing healthcare.gov and released the same PoS that was actually released? It'd be even worse than the current fallout because they aren't subject matter experts.
At least OR appeared to try and be competent by having an in-depth sales cycle and outside consultants to vet the proposals.
I'm sure they could do this project with OSS products and custom code integrating everything, but those don't come with support contracts. If there's no support contract that means OR has to support it itself, which means hiring people with deep knowledge of the system to quite possibly do nothing for long periods of time until a bug is found for their particular use case, because no one else is using the product.
That's not cost effective compared to an off the shelf solution from a vendor that can use 1 support staff to help multiple customers, especially when you realize this means requiring at least a security expert, a db expert, and a frontend UI/UX.
It's exactly the same reason why tech companies don't write everything themselves from OSS projects. Sure, they do that for certain things that they deem critical, but not for things like salesforce or the direct deposit and accounting software and sometimes not even for stuff as important and tech-ey as a vcs, like perforce.
I'll totally agree that OR should keep IT in house for this stuff, but I don't think it's a great idea for a state or business to suddenly get NIH syndrome and want to write custom software themselves for anything their agencies might need. Can you imagine if this had happened because OR did this project in house and had the same results? Or if the whitehouse had said they weren't outsourcing healthcare.gov and released the same PoS that was actually released? It'd be even worse than the current fallout because they aren't subject matter experts.
At least OR appeared to try and be competent by having an in-depth sales cycle and outside consultants to vet the proposals.
Yeah, that's due to a strong bias away from anything else. Lots of lobby dollars in play reinforcing that, and there is the strong perception that Oregon can't actually afford to make any kind of investment.
But this shows Oregon can, did, and it was botched horribly. Doing the same while providing local jobs, some great skills to be used all over the place, and build something that has a very likely lower overall ownership cost profile makes a ton of sense!
It just won't get proposed in the current political climate, which was the point of my reply.
Of course Oregon wants COTS. Everybody does, because they think the vendor will own it, saving them money through economies of scale. Quite frequently, COTS can do that, but for something like this, something new, that needs to communicate with a lot of varied systems?
No way in hell it was ever gonna be COTS, and that right there should have triggered a much different, "how does Oregon do this?" kind of discussion.
Oracle has a good sales team, and they sold it right to this dynamic, which is what they were supposed to do. Problem is the value really wasn't there. No matter what, it was going to take some real work to build out the exchange and have it function well.
As for the support contracts, those come with an annual cost. It's very highly likely that cost could simply provide a job for the people who built the thing. Additionally, having seen success on a project like that, the same team would be well positioned to build out more and or get funded by other municipalities seeking similar solutions.
I don't have the basic numbers a few of us ran back then, but they were compelling.
This works the same for States as it does companies or anyone else. Make the in-house investments and own the tools. From there, buying help is always an option. Avoiding it is another option. Relying on a vendor, who will very frequently oversell their support staff, counting on their product to be a winner, often results in that support contract merely funding the trouble shooting. More dollars are required to buy, implement, test, etc... fixes.
Fixes, which by the way, will be called custom patches for major customers due to the size and non COTS nature.
But this shows Oregon can, did, and it was botched horribly. Doing the same while providing local jobs, some great skills to be used all over the place, and build something that has a very likely lower overall ownership cost profile makes a ton of sense!
It just won't get proposed in the current political climate, which was the point of my reply.
Of course Oregon wants COTS. Everybody does, because they think the vendor will own it, saving them money through economies of scale. Quite frequently, COTS can do that, but for something like this, something new, that needs to communicate with a lot of varied systems?
No way in hell it was ever gonna be COTS, and that right there should have triggered a much different, "how does Oregon do this?" kind of discussion.
Oracle has a good sales team, and they sold it right to this dynamic, which is what they were supposed to do. Problem is the value really wasn't there. No matter what, it was going to take some real work to build out the exchange and have it function well.
As for the support contracts, those come with an annual cost. It's very highly likely that cost could simply provide a job for the people who built the thing. Additionally, having seen success on a project like that, the same team would be well positioned to build out more and or get funded by other municipalities seeking similar solutions.
I don't have the basic numbers a few of us ran back then, but they were compelling.
This works the same for States as it does companies or anyone else. Make the in-house investments and own the tools. From there, buying help is always an option. Avoiding it is another option. Relying on a vendor, who will very frequently oversell their support staff, counting on their product to be a winner, often results in that support contract merely funding the trouble shooting. More dollars are required to buy, implement, test, etc... fixes.
Fixes, which by the way, will be called custom patches for major customers due to the size and non COTS nature.
It could go badly either way.
However, I find it very hard to believe a competent group would actually fail in this spectacular of a way.
Vendors often know just a little more than the client does, and they do oversell, and "cost effective" gets calculated to a point where the purchase price and services costs are much closer to building something than not.
And some of my career is in selling just that kind of enterprise solution, and that's exactly how it goes a lot of the time. There is value there for sure, and the vendor will need to capture as much of it as it can in revenue, while delivering as little service as possible to keep margin.
Those expensive people, who likely would have built the system, wouldn't be doing nothing. Not at all. Once that system is up and successful, they continue to build lots of things OSS style. Boring, simple, cost effective things and anybody anywhere can pick them up and use them too.
Lots of synergies here for States and municipalities looking to create jobs and manage costs.
Having been on the ugly end of these things, it's worth a look. Perhaps it wasn't the right call for this project, but it really wasn't evaluated. And there are few entities able to answer an RFP to make that evaluation, who don't also sell very expensive software licenses.
Should have been evaluated, and Oregon should have some basic competency in State IT to do that. The savings potential is significant, and Oregon needs to own it's IT anyway.
For a lot of things, vendor supplied software will win out. No question. But for those things where it won't, or cost of ownership over long periods of time do not make sense for the public, having a means to explore that was what the legislation was all about.
Keeps vendors a bit more honest too.
However, I find it very hard to believe a competent group would actually fail in this spectacular of a way.
Vendors often know just a little more than the client does, and they do oversell, and "cost effective" gets calculated to a point where the purchase price and services costs are much closer to building something than not.
And some of my career is in selling just that kind of enterprise solution, and that's exactly how it goes a lot of the time. There is value there for sure, and the vendor will need to capture as much of it as it can in revenue, while delivering as little service as possible to keep margin.
Those expensive people, who likely would have built the system, wouldn't be doing nothing. Not at all. Once that system is up and successful, they continue to build lots of things OSS style. Boring, simple, cost effective things and anybody anywhere can pick them up and use them too.
Lots of synergies here for States and municipalities looking to create jobs and manage costs.
Having been on the ugly end of these things, it's worth a look. Perhaps it wasn't the right call for this project, but it really wasn't evaluated. And there are few entities able to answer an RFP to make that evaluation, who don't also sell very expensive software licenses.
Should have been evaluated, and Oregon should have some basic competency in State IT to do that. The savings potential is significant, and Oregon needs to own it's IT anyway.
For a lot of things, vendor supplied software will win out. No question. But for those things where it won't, or cost of ownership over long periods of time do not make sense for the public, having a means to explore that was what the legislation was all about.
Keeps vendors a bit more honest too.
> I'm sure they could do this project with OSS products and custom code integrating everything, but those don't come with support contracts.
I'm sure that if you pay enough money (and considerably less than what they threw at Oracle), you can get support contracts for all kinds of open source software and custom integration solutions.
That's the beauty of open source: you don't have to be the vendor to support it.
I'm sure that if you pay enough money (and considerably less than what they threw at Oracle), you can get support contracts for all kinds of open source software and custom integration solutions.
That's the beauty of open source: you don't have to be the vendor to support it.
The main problem is that there is no such thing as 'a COTS solution for a water billing system with little custom code'.
The only thing that exists is what companies like Oracle and SAP market as such, claiming their solution only requires 'configuration'. They confuse their customers with misuse of terminology, because in fact that 'configuration' consists of a lot of custom code that is needed to adapt their solution to a specific purpose.
Whether it is more or less code than honestly and transparently assembling such a system from available OSS components is besides the point as long as customers are not crystal clear on the fact that COTS solutions rarely exist, except as components for a purpose-built system.
A claim of having a COTS solution is almost always a sham for any reasonably sized project.
The only thing that exists is what companies like Oracle and SAP market as such, claiming their solution only requires 'configuration'. They confuse their customers with misuse of terminology, because in fact that 'configuration' consists of a lot of custom code that is needed to adapt their solution to a specific purpose.
Whether it is more or less code than honestly and transparently assembling such a system from available OSS components is besides the point as long as customers are not crystal clear on the fact that COTS solutions rarely exist, except as components for a purpose-built system.
A claim of having a COTS solution is almost always a sham for any reasonably sized project.
I agree with you on "tried", but this isn't the first State failure with that method, nor the first one with a lot of digits attached to it, though this one is the biggest I know of.
Last time I did this, I hired a subject matter expert, or was that expert. Then the interests are clear, and it's not just a paid service. Oregon really should bring that competency into the State proper so that this kind of thing gets owned, not just sourced and managed.
Do that with whatever software is best indicated too, but own it. Oregon didn't do that, again...
Last time I did this, I hired a subject matter expert, or was that expert. Then the interests are clear, and it's not just a paid service. Oregon really should bring that competency into the State proper so that this kind of thing gets owned, not just sourced and managed.
Do that with whatever software is best indicated too, but own it. Oregon didn't do that, again...
> Enterprise sales will get just a notch tougher real quick.
For Oracle, or in general?
In an ideal world, if Oregon prevail, and if the message was delivered to every vendor whose salespeople blatantly (and knowingly) lie in order to win business, then Enterprise Sales would get considerably easier for those salespeople who prefer to sell based on substantiated fact.
Purchasing would also get easier for organisations trying to sort the wheat from the chaff in RFP responses.
Sadly I think this is unlikely to ever happen, regardless of the outcome. The message is unlikely to spread, and even if it were to, some (many?) enterprise software and hardware vendors have a sales culture that will not easily be changed. Oracle amongst them.
For Oracle, or in general?
In an ideal world, if Oregon prevail, and if the message was delivered to every vendor whose salespeople blatantly (and knowingly) lie in order to win business, then Enterprise Sales would get considerably easier for those salespeople who prefer to sell based on substantiated fact.
Purchasing would also get easier for organisations trying to sort the wheat from the chaff in RFP responses.
Sadly I think this is unlikely to ever happen, regardless of the outcome. The message is unlikely to spread, and even if it were to, some (many?) enterprise software and hardware vendors have a sales culture that will not easily be changed. Oracle amongst them.
I don't know. Maybe in general, depending on how this all goes down.
For Oracle specifically, it's gonna get a notch harder. Every single one of their peers is going to get that info and use it. Competitive info is often shallow and it gets tossed about frequently. (though doing that often does the ones using it more harm than good)
You are right about it getting somewhat easier for the ones actually selling on facts and material value. That's a much harder sell, and I find qualification to be particularly vital, because the level of effort required to vet the deal for everybody isn't trivial. And these things need a good vetting.
The easy sell is to promise it, take the money, then toss it into services, who will somehow magic it up while maybe breaking even. Never goes well, or if it does, it's an investment that pays off with annual contracts.
The harder one is to establish value, metrics, requirements, etc... up front, figure out what those could be worth, work a little down from there just to be safe, then sort out who does and owns what. Sell one of those, and it's probably a winner.
For Oracle specifically, it's gonna get a notch harder. Every single one of their peers is going to get that info and use it. Competitive info is often shallow and it gets tossed about frequently. (though doing that often does the ones using it more harm than good)
You are right about it getting somewhat easier for the ones actually selling on facts and material value. That's a much harder sell, and I find qualification to be particularly vital, because the level of effort required to vet the deal for everybody isn't trivial. And these things need a good vetting.
The easy sell is to promise it, take the money, then toss it into services, who will somehow magic it up while maybe breaking even. Never goes well, or if it does, it's an investment that pays off with annual contracts.
The harder one is to establish value, metrics, requirements, etc... up front, figure out what those could be worth, work a little down from there just to be safe, then sort out who does and owns what. Sell one of those, and it's probably a winner.
The state is not just suing for breach of contract to recover some money here -- the Attorney General is seeking a permanent ban on Oracle selling anything to the state.
"On page 119: (c) A permanent injunction prohibiting Oracle from marketing to or entering into a contract with any public corporation of or agency of the State of Oregon from the date of judgment forward. "
"On page 119: (c) A permanent injunction prohibiting Oracle from marketing to or entering into a contract with any public corporation of or agency of the State of Oregon from the date of judgment forward. "
Wow.
I'd be surprised if there wasn't a "public corporation of or agency of the State of Oregon" that currently uses some piece of Oracle software. I'm sure current contracts wouldn't be affected, but what about when they come up for renewal?
I'd be surprised if there wasn't a "public corporation of or agency of the State of Oregon" that currently uses some piece of Oracle software. I'm sure current contracts wouldn't be affected, but what about when they come up for renewal?
You could well be correct, but in that case it's an opportunity to take a hard look at how adequately the "system" used by that agency is actually working, and possibly make revisions where indicated or necessary.
Trust me, I know there are Oregon state agencies that have suboptimal software systems. Even though Oregon is a low population state, it still runs a large number of systems, and among them are others that are bound to be marginal.
It's silly to be idealistic, nonetheless it's denying the obvious to miss the chance to "do it right", to develop and apply the rigorous process alluded to in several comments above.
A plan for the long haul, integrating and unifying systems, and reducing "reinvention of wheels" where appropriate, and organizing truly knowledgeable oversight of the state's data management would go a long way to prevent recurrence of recent mistakes, have systems that work better, and cost the public less.
No one has to tell me it isn't going to happen, Oregon is where I live, and logic doesn't stop me from wishing that it would.
Trust me, I know there are Oregon state agencies that have suboptimal software systems. Even though Oregon is a low population state, it still runs a large number of systems, and among them are others that are bound to be marginal.
It's silly to be idealistic, nonetheless it's denying the obvious to miss the chance to "do it right", to develop and apply the rigorous process alluded to in several comments above.
A plan for the long haul, integrating and unifying systems, and reducing "reinvention of wheels" where appropriate, and organizing truly knowledgeable oversight of the state's data management would go a long way to prevent recurrence of recent mistakes, have systems that work better, and cost the public less.
No one has to tell me it isn't going to happen, Oregon is where I live, and logic doesn't stop me from wishing that it would.
You know, quite a lot of people running Oracle are dreaming of replacing it with Postgres.
(At the day job, we're finally proceeding with this next year. I'm sure there will be many a tale to tell!)
(At the day job, we're finally proceeding with this next year. I'm sure there will be many a tale to tell!)
[deleted]
my favorite bit so far:
> He added that “they broke every single best practice that Oracle themselves have defined. It is one of the worst assessments I have performed * * .” The same developer wrote to Oracle, “You are Oracle people, working on an Oracle hardware platform, with Oracle technology products, on an Oracle solution. * * Oracle should be delivering these environments and products as a solution, like they actually understood the products and owned the solution which has not been the case by a long shot.”
> He added that “they broke every single best practice that Oracle themselves have defined. It is one of the worst assessments I have performed * * .” The same developer wrote to Oracle, “You are Oracle people, working on an Oracle hardware platform, with Oracle technology products, on an Oracle solution. * * Oracle should be delivering these environments and products as a solution, like they actually understood the products and owned the solution which has not been the case by a long shot.”
Makes sense in theory, problem is the software (Fatwire) was acquired, and the talent was "body shop" quality subcontractors. Oracle's database product is really the only thing they've built themselves and understand, everything else has been purchased.
Where's the case of the taxpayers of Oregon against the state that due to sheer negligence and ignorance managed to deliver nothing?
They did not oversee the work, they allowed Oracle to oversee itself, and they hired outside consultants at every single fucking point along the way to do what was their job.
They did not oversee the work, they allowed Oracle to oversee itself, and they hired outside consultants at every single fucking point along the way to do what was their job.
A former Oracle employee was more direct saying Oracle’s
products were seemingly configured “by a kindergartner.”
&
An internal assessment estimated that the “Oracle Solution” was 40% custom code
Anyone know who the kindergartners were ? I believe Oracle are now in the business of sending their software development to far away sweatshops.
I'm pleased that action is being taken here - software now seems to be seen as a commodity by far too many - the craftsmanships is gone and the salesman have taken over.
&
An internal assessment estimated that the “Oracle Solution” was 40% custom code
Anyone know who the kindergartners were ? I believe Oracle are now in the business of sending their software development to far away sweatshops.
I'm pleased that action is being taken here - software now seems to be seen as a commodity by far too many - the craftsmanships is gone and the salesman have taken over.
Fascinating view of yet another high-profile software project failure. Also a demonstration of the consequences of not making getting software development right a priority.
I guess it is easier to blame Oracle than to fire the truly incompetent managers of this project.
A fool and his money are easily parted
Oh, I know why. Oregon actually had a bill presented in the early '00's to accomplish that kind of thing. I was part of it, and it was all about making sure the State could evaluate Open Source type solutions right alongside proprietary ones.
It failed to see a vote, despite broad bi-partisan support, due to some expensive suits lobbying the speaker Minnis.
One case in point used then was the Portland Water billing system. That one cost a ton of money, and was released as a failure, costing more money, and in the end, is likely still costing money.
For this kind of change, Oregon could very easily have setup a foundation, or working group of some sort, and had this thing built out of many, if not all OSS components. Oregon would own it, and have more control over it, and would be able to exercise a lot more options managing it too.
Once that group saw success, they keep the jobs, Oregon saves cash, and they then start knocking out municipal solutions of all kinds, exporting that as needed for funding and or just cash for the public interest.
Oracle really screwed the pooch on this one. It's going to be painful for all involved, but I sure wouldn't want that document out there floating around. Enterprise sales will get just a notch tougher real quick.