So far the state of Oregon has paid Oracle at least $43 million dollars for a website primarily intended for residents to use to enroll for healthcare insurance under the Affordable Care Act. The site was initially due for delivery on October 1, which now has been pushed back on numerous occasions because the site doesn’t work. In fact, by all accounts, it’s an outright disaster.
The most recent missed deadline was announced on December 5, according to a report from KOIN TV.
“Cover Oregon’s former director, Rocky King — who took a long-term medical leave of absence last week — had said the online system would be ready Monday for insurance agents and community groups that have contracts with Cover Oregon, and then Dec. 16 for all individuals. King was replaced by Goldberg, who is the director of the Oregon Health Authority…
“Both those deadlines will be missed. In an email to KOIN Monday, Cover Oregon spokesman Michael Cox alleged that the Dec. 9 and Dec. 16 dates were ‘targets set by Oracle, not deadlines set by Cover Oregon.'”
This last missed deadline would have occurred after the site would be of any use for Oregon residents seeking medical coverage for 2014. It’s now estimated that the site won’t be operational until at least the end of January. Meanwhile, the state has scrambled to hobble together a labor intensive workaround using paper forms which are being processed manually, requiring an additional expenditure of $4 million for 400 temporary workers.
Inquiring minds want to know why. Why Oracle? Why so much money?
I’m no Malcolm Forbes when it comes to money nor am I any kind of hotshot coder. To try to wrap my head around the situation, I arranged for a telephone conference with Josh Barratt, who’s the Chief Architect for the upscale hosting company Media Temple. It turned out he wasn’t privy to any inside information, but his years of experience working on large website projects meant he had plenty of insights to offer.
“Oracle wouldn’t be my first choice but it sounds like they put it up for a bidding process and when everyone saw how complicated the project was, Oracle’s the only one that hung in there.”
Indeed, the site is complicated. Maybe too complicated. Many press reports have mentioned the ambitiousness of the project.
Unlike other states that were happy to merely have a website to help guide their residents through the unknown territory that is the Affordable Care Act, Oregon wanted to put all of their health care eggs into this one Internet basket. In addition to insurance for individuals, they wanted the site to handle business health insurance as well as Medicaid and just about every other interaction a resident may need to have with the government concerning healthcare. That level of complexity is part of the reason why the site has cost so much money and one of the reason’s why it’s not yet operational.
The other reason is Oracle.
The folks from Larry Ellison’s Redwood City fiefdom understood the enormity of the project from the getgo. They also knew to expect more complexity to be added as development was taking place, especially since federal regulations were still in flux when they took on the project. This being the case, one might assume they would put their best people on the job, which doesn’t seem to be the case.
“It’s hard to judge the quality from the outside, but if you look at their front end website, there are a lot of things they didn’t do. When you look at their technology practices, even the ones we can see from the outside, it’s obvious they didn’t put their ‘A’ team on this. So, yeah, I do think that Oracle didn’t do a good job and they probably shouldn’t have taken it in the first place, because they’ve taken on something it would be impossible to do a good job on.”
This would seem to be a very important cautionary tale for anyone considering hiring Oracle for a major project. They’ve already taken at least $43 million from Oregon, with some sources putting the figure at closer to $90 million, yet they don’t bother to send their best people to design and build the project. Is that any way to treat a $40 million dollar client, even a government one?
It’s unclear whether Oracle stands to continue to profit on the Cover Oregon site after it’s operational, from licensing fees, long term support and the like. Although it appears as if Oregon will end up with ownership of the custom code being written specifically for them, we can probably assume that the site utilizes components from the proprietary Oracle stack which will, at the very least, require licensing and support contracts, perhaps in the neighborhood of millions annually.
That’s if a fully functional Oracle built site ever goes live. Right now, it would probably surprise no one if the state decided to drop the Oracle project altogether and find another solution. In September, Cover Oregon quit making payments to Oracle and is currently sitting on over $18 million in Oracle invoices.
According to Media Temple’s Barratt, this project would be quite expensive even if costs weren’t being dictated by Oracle’s inflated price list. Not only is the site unbelievably complex, the fact that it concerns healthcare bring a whole host of other issues to the table.
“There might be regulations which can raise your cost because you have to do things in a more careful, deliberate way. You have to have people that have the expertise to even know what those regulations are–in how you’re going to get the data and so forth. It’s not as simple as getting a bunch of programmers in a room and paying them a reasonable wage and pizza. You need to get people that are trained more. At least get people who can be training as well. People who know how to build high traffic web sites.”
That being said, however, the money that Oregon is shelling out is still incredibly high, even for a project of this magnitude. Said Barratt, “If somebody had just built a team around this, they probably would have spent $40 million in the first ten years.”
Part of the problem, Barratt said, is that government agencies “don’t think like start-ups.” As a general rule, they’re headed either by career civil servants or by elected officials, often with little or no business background. This can lead to decisions that even the most inexperienced businessperson probably wouldn’t make, especially when under the seductive spell of a well trained sales crew.
“The contract was badly written. It was basically just time and materials. There were no significant delivery penalties. Basically, Oregon paid Oracle to work really hard on this project, but they didn’t pay them really to finish the product, which is the alternative way that you can structure a lot of these contracts. There’s really no downside to Oracle if they don’t ship.”
Even though Oregon will evidently end-up with ownership of the code written specifically for this project, it’s hard to imagine that the site will utilize open source components such as databases, as Oracle has never met an open source project it likes, even those they own. However, if the site were built using existing open source components, tied together with custom code to make everything function as needed, Oregon would truly own their site, which they could release under the GPL or some other open source license. Barratt points out that this would bring many far-reaching benefits.
“Especially for a state exchange type thing like this, where there’s multiple states solving the same problems, it’s even better if they build it in open source because you can get community feedback on your practices and other states and government agencies can leverage your work and collectively save even more money. I’m very fond of the idea that all taxpayer funded code development should be open source, because it’s my code and it’s your code. It’s our code.”
For now, though, Oregon is stuck with a very expensive white elephant and most of its residents will not be able to take advantage of the benefits of the Affordable Care Act until 2015.
Christine Hall has been a journalist since 1971. In 2001, she began writing a weekly consumer computer column and started covering Linux and FOSS in 2002 after making the switch to GNU/Linux. Follow her on Twitter: @BrideOfLinux
I wonder if a many companies looked at the requirements and decided to pass because they realized the project would likely be a disaster. Thus, they decided that their corporate reputation was with more than the profits from the contract.
Many of the major contractors in the whole affair have taken a serious PR hit. Effectively they have been cast as incompetent morons who should not be trusted with any project whether it is fair or not.
The article noted the contract had no real penalties for Oracle. Apparently, government entities are not equipped to properly design and manage a large, custom IT project.
And I’m willing to bet the specs changed every hour/day/week. A small change to a complex web site may involve hundreds if not thousands of hours of lost labor and anomalies nobody expected or could reasonably foresee. It would be interesting to hear Oracle’s side. I’d bet they have all their T’s crossed and i’s dotted in change orders. It seems to be the press that is complaining, not Cover Oregon.
@A Linux Lurker Not true. KBOI TV in Oregon has unearthed a host of internal emails from CO that deal with this issue. Here’s a summary of just a few:
The state’s top politicians – including Governor John Kitzhaber and Sen. Jeff Merkely – were so alarmed they bypassed King and contacted Oracle executives and Cover Oregon staff directly to understand the problems.
The problem, they heard, was Oracle.
Cover Oregon’s emails – both internally and those sent to Merkley’s office – describe Oracle’s performance as “at best, unacceptable.”
Oracle’s development team had an “undeserved amount of arrogance” and “their overall technical skill levels are questionable,” the emails show.
In short, Oracle “did not have the in-house expertise for what they were contracted to deliver.”
If you like, you can go to http://www.kboi2.com/news/local/New-emails-show-Cover-Oregon-unraveling-days-before-launch-236021451.html where you’ll find links to screen shots of many of the emails. Remember, CO is now refusing to pay Oracle any additional funds and are sitting on top of at least $18 million in dunning invoices from Oracle.
Also, almost every techie who’s been called in from outside of Oracle to try to fix the site has commented publicly on the poor quality of the code.
Oracle may have damaged their reputation by two actions:
1. Getting involved a project that was likely to troubled. Many massive IT projects suffer from being to big and complicated. The project violated the KISS meme of being focused on solving one problem at a time. This is the fault of the Oregon ultimately for trying to do too much though Oracle did not have to bid on it. Even fairly “simple” projects can have hidden complexity that is not apparent until one starts detail design and code writing.
2. Apparently failing to execute their side of the contract properly. The goal is to be a position when the finger pointing occurs to be able to defend yourself. I do not know what the paper trail will show here but I suspect this will end up in court with a messy lawsuit. I do not see Oracle (Larry Ellison) backing down because Oregon is upset.
I think many of the project lead contractors on the various healthcare websites involved will suffer because there is a perception of incompetence combined with greediness.
The news reports imply that primary contractor for the federal website was at best marginally competent to do the work. Coupled with managerial incompetence within the government a disaster was waiting to happen.
You’re probably correct. Since Nancy hasn’t bothered to read the bill yet most of us don’t even know what is in it yet. How can ANYONE develop a web site that coordinates with the federal web site (in it’s condition), the IRS, the DOL, the SSA, the DMV and who knows who else with specs that are probably changing daily if not weekly due to government incompetence from the U. S. Senate and House right down to the local level and bureaucrats in management who have no clue what technology actually is let alone have a clue how to write a spec for implementation of Obamacare. One thing is for sure: Oracle didn’t create the products they build without first rate engineering so that complaint is, on its face, a bold faced lie to cover up government manager incompetence.
Until the trial starts and the evidence is presented this whole fiasco is speculation regarding a probable comedy of errors. I wouldn’t give ANY weight to internal emails from people attempting to weasel out of their incompetence. Nor would I give any credence to a “techie” who doesn’t like the format of the HTML, pathetic as it is in its own right.
It will be interesting to see the evidence emerge from the trial. And I wouldn’t place any bets that the CO management provided a comprehensible spec.
It boggles the mind to know that first the federal government, specifically the Medicare agency with the Healthcare.gov website and now the Oregon State government with their Oracle? developed health care website would not seek out world class expertise in this field that has proven ultimately successful and reasonably costs as compared to the outrageous fees paid out by these two doofus governments to incompetents
There exists a comprehensive healthcare type related Web portal – https://osha.europa.eu/en – the European Union Agency for Safety and Health at Work, that serves over 400 million European Union citizens in 18 languages for 27 countries.
Furthermore, the core software under-pinnig the entire infrastructure for this project is Free/Open Source Software (FOSS) named Plone, which co-incidentally also powers web portals for individual European Countries’ governments, all the web portals for the Brazilian government, major universities in USA, UK, Europe, South America as well as the AARP, US CIA and FBI agencies’ primary portals (for it’s vastly superior reliability and especially security) and many dozens more web portals that effectively operate with equal or greater volume and stress than these two stupid US government web portals.
Why is it an addiction for so many US government departments and in Congress itself to throw hundreds of time more money at projects run by “big name” commercial entities who are popular and major political contributors, rather than take a more pragmatic and proven approach?
Crazy, idiotic thinking. No wonder the populace are totally disenchanted with politics and politicians.
Oregon should consider hiring consultants that will develop the new website on open source alone. Build it on a proven platform like Red Hat Enterprise Linux for example. That way Oregon can maintain a group of its own employees to maintain the website. Oracle can develop the site, leave, and Oregon will be scrambling to maintain it.
W Anderson, very well put! I would bet you have seen similar situations like this before.
I’ve worked for a major financial firm that had been joined at the hip with Oracle since I’d arrived and I can tell you if Oregon’s experience with Oracle is anything like mine the project was doomed the moment the contract was signed. You see Oracle may have engineered a world class RDBMS back when they were young and hungry, but these days Oracle only about marketing and sales. Oracle sales executives sold the company I worked for the moon and once the contracts were signed and invoices paid they delivered Shinola. Through back channels (read golf packages and all expense paid trips to Oracle World marketing bonanzas for executives) Oracle gave a wink and a nod to the idea of porting the company’s Sun identity management solution to their new platform. 9 months into the project we discovered that not only did the tools Oracle provided for the migration mostly not work, but those that seemed to work left the data corrupted and generally useless. After contacting Oracle repeatedly and opening trouble tickets left and right it became painfully clear that Oracle did not have the expertise in house to get the job done and had been stalling presumedly trying to hire back some of the Sun engineers that they’d just acquired and let go. Never did Oracle come clean and admit that the solution they sold not work as promised, but they constantly pushed it back on us having us provide log files, stack dumps, install patches that didn’t work, and anything imaginable to say they were waiting on us. I will never take a job working with Oracle products other than their core database again.
There’s one major point that was forgotten. Oracle is not a company of web developers. That’s not their bread and butter.
@Eestin – The front-end website shouldn’t be the issue because all the business rules and logic should have been clearly defined most of which affects back-end processing with only a minority of the business logic directly impacting the data-entry front-end web form(s).
I have seen the same incompetence from Oracle, Sun, and a variety of other “consulting houses.” One project that I took over from a Sun certified expert (cough, cough) employed by Sun Microsystems was utterly useless and their work product contained explicit references to another client. After the implementation work was dumped in my lap it was necessary to restart from scratch yet I completed a year worth of work in less than three months.