State data center: $92 million but cloud services are extra??!!
The Department of Information Services Request for Proposal (RFP) for the equipment, consulting and infrastructure of the state data center has hit the agency’s website here. It’s pretty well written and some genuine thought has gone into the content. I congratulate the DIS folks who worked to make this a strong product. It’s not an easy effort and I really appreciate that reality.
I haven’t had time to really review it other than a cursory look. A few quick thoughts with more to come once I’ve done a deeper dive. (And let me know if you find things worth exploring).
Folks tell me the budget for the internal projects listed appears to be $92,225,445. No, that’s not a typo. $92 million.
Breaking it down: $54 million is currently planned for the move and set-up of the data center networks. But from what I can tell it seems that a vast, overwhelming majority of equipment is (as I suggested on KUOW) the SAME mainframe system the state already owns and uses. That is, those systems that don’t interoperate and have old, trapped data that can’t be used for new services. Here’s a recent California example of the same problem.
So what that means to taxpayers and citizens is that at the end of this entire adventure is that the state’s same old databases, with the same middleware and applications, will have the same functionality, features, capabilities and limitations as today. I have to wonder what the opportunity cost of this project really is for our state’s technology strategy….world-class technical training for DIS employees…new mainframes, middleware, applications for agencies….a coordinated approach that uses the best of the private sector and the value-add of the public sector….and so much more.
As I previously reported, $15 million will be spent on router (switching) infrastructure itself.
And the writing in the RFP is on the wall: Cisco is now the official standard for routers and switching in the State of Washington without (I suspect but have not confirmed) a competitive bid process.
If DIS went through an aggressive competitive bid process prior to selecting Cisco, I offer my sincere and public apologies. If anyone other than Cisco provided the architectural roadmap for how many Cisco routers we need to buy, and how to design the system, then it would be interesting to compare that technical schematic with the one Cisco provided. Beyond the data center, I hope a business decision that ultimately will be in the hundreds of millions as we replace routers statewide would warrant at least an independent, second technical opinion.
You get the idea.
Still, from a fiduciary perspective outside of technology, I find it troubling that an agency may have made a unilateral policy, financial and business decision to go with one vendor of a commodity product without a competitive process. This is a policy decision of extraordinary proportions and serious financial and technical implications. Cisco’s pitch is always that they are “best of breed.” That was a great sales pitch in the 1990s. Why not go with two vendors and put the heat on to keep prices low and service levels high? We just set ourselves up for a very expensive monopoly relationship.
Equally important, if Cisco is now our exclusive provider, did the Information Services Board (ISB) make this policy, financial and technical decision or was it made by the agency alone?
Now here’s the strangest part so far:
From what I can tell, there is NO actual equipment in the $92 million to pay for the ‘public cloud computing services’ that DIS now says is one of the central justifications for the entire data center project. It means DIS will manage and operate ‘cloud computing services’ for all of state government as well as other levels of government in Washington.
But a technical buddy who knows Cisco tells me that meaningful cloud services probably require Cisco’s fancy VBox solution with EMC and VMware. Nothing in the RFP or other materials that I can find talk about that type of sophisticated equipment.
Does DIS really think it’s going to ask the Legislature for more money to deliver the actual product and service they used to help justify the $300 million investment?
One of my top goals during the 2010 Legislative Session is to find $1.4 million to restore funding that has been completely eliminated in the budget for academic tutoring, mentoring, counseling and educational advocacy for youth in foster care. The Treehouse employees who serve these foster youth are scheduled to be laid off December 31 even before the Legislature has a chance to decide for itself.
DIS, can you spare a dime?
Your partner in service,
Reuven.
Sir (and I use that term a bit loosely here) you speak from an obvious position from someone who is severely lacking in knowledge. I personally offended that you have suggested that personal from a state agency would intentionally commit an act of fraud.
You appear to be what has been traditionally known as a muckraker (for those not familiar with the term – To search for and expose misconduct in public life.). Yet you do not know much about the technology industry. If you read thru the link that you presented for the RFP, you will see that the old adage of “it is in the details” applies here. I am a 30 year veteran of the computer industry and have worked both in the public and private sector throughout my career and for the most part people in this industry are painfully careful about these kinds of requests.
In respect to “your buddy” that knows Cisco, I suspect you try asking a real professional who actually works in this business about what the needs are and about looking at what the department actually owns BEFORE you suggest that they need additional items. Let me point out a few things to you from an actual professional (ME).
First the term cloud computing is a brand new term, what it means and how it will applied is highly depended on more than a very few pieces of hardware and software, you have no idea what is currently in the network of this department and you have no idea what is in the pipeline of this department I again I take offence to you making such a blatant statement without actually asking someone about it. Moreover, your constituent’s now have a reason to question whether or not they made a proper choice in you as a person to represent them.
You are fortunate in the fact I do not live in your district because I would be out not just campaigning for your opponents but actively attempting to have someone like yourself removed from office for speaking out without actually looking into the facts (we know what they are but you apparently do not understand how to find them without checking first). As a citizen I am appalled at your behavior.
Teddy,
Wow, some sharp feelings there? Sorry I may not be technically competent (true indeed) and sorry you feel the people of the 36th Legislative District are poorly represented. Given that I won 66%-34% and 94% of the precincts in my district I respectfully suggest the people of my community might disagree with you.
It would be extremely helpful to my learning curve if you as a DIS employee would say something of substance technically, and explain where/why/how my analysis is inaccurate.
If you will provide specific examples of statements that I made that are factually inaccurate I’ll gladly clarify and apologize if necessary. The obligation of public officials is to raise tough issues in a Democracy, not to put our heads in the sand and bend to the technical (or military, etc.) experts who try and intimidate elected officials into writing blank checks.
I think highly of the DIS folks. You seem pretty offended that I’ve taken these issues to task, but I believe most DIS employees are passionate believers in their work and in their role in building quality of life in our state. I challenge you Teddy as a DIS employee to engage with me to help improve the quality and efficiency of how we spend our technology money in Washington. I believe it’s time that DIS and technology in our state was more transparent, open with vigorous and healthy debate. Should we outsource email? Should we consolidate purchasing? Should we use the cloud? Should we change the cost recovery model for DIS so you are able to provide higher levels of service and not just cost driven? I’m raising issues that I believe your colleagues are raising around the water cooler because they care about high quality services.
Challenging assumptions, business models and processes, contracts and decisions is not anti state employee–it’s pro government!
We’re in this together.
Your partner in service, Reuven.
Reuven,
Thank you for your service and active pursuit of excellence for the citizens of Washington State. I see that you have a healthy interest in the new State Data Center in Olympia and for good reason. There are a lot of decisions that must have been made in order to come up with what is being built there as we type.
I ,personally, am glad to see the State investing in our future and that of our children. I do not know all the strategies involved in determining what will be installed into the State Data Center but I trust that the State Employees have the State’s best interest at heart and will make the right decisions. We hired them to do the job, let’s let them do it and quit second (or hundredth) guessing their every move. They are good people and great representatives of the best and brightest, leading our State into the future of “Cloud Computing”. Let’s cheer them on as they spend countless hours to bring us all into the modern world with all the efficiencies that come with it.
Hire good people, set a goal, lead from the front and watch out for success. This is the recipe that we need to adhere to.
I don’t see anything constructive to come from being the consumate micro managing second guesser. Trying to gain political traction in a public forum at the expense of the reputations of so many excellent State employees that have labored so long is unjust and unworthy of the Representative of the City of Seattle.
I hope you continue to ask questions, but in private and with those that have diligently looked after the spending of taxpayer dollars.
Thank you
So, I have been watching this issue from afar, and am wondering, where it is at? I see that the State of Washington Legislative session has started and that the only recent news on this data center project is from the Olympian, which only plays a cheerleader for the job creation and the “getting along with the neighbors” aspect of the building. The Department of Information Services seems absolutely quiet, and I fear that this building will have created a temporary boon of jobs, but the result will cost the taxpayers of Washington in the long run.
Hopefully, there will be some vetting of this in the legislature and that the Department of Information Services will provide some insight into their plans because as of right now, they seem to be building a large toilet with which to flush the taxpayers dollars.
Rueven,
Thank god for you! One person brave enough to ask for real answers. I’ve read your blog and followed the NPR reports…, then checked out the State Auditors review and detailed appendices (which were not easy to find by the way)…; the detail starting on pg. 74 of the final report put’s an interesting twist the whole discussion….
https://www.sao.wa.gov/EN/Audits/SGPR/Documents/Final_TPI_report_Web.pdf
These pages seem to indicate that of the agencies studied, nearly all are fairly competitive when compared to the private sector…, according to TPI’s Mark to Market comparison. And some are extremely competitive. Where it get’s interesting is when the Department of Information Systems (DIS) gets involved…, the prices are rediculously high.
And as I understand it; it is DIS that is sponsoring the need for the new data center to consolidate services and save mountains of money. From the looks of the Auditor’s study, DIS is the last agency you would want to lead this effort. Unless they think that the DIS costs are representative cost to provision IT services within state govt. (swing and a miss). Do the other agencies think this is a good thing?
Back to consolidation and a new data center for the state. I know that the technologies to enable consolidation are there and Cloud computing offers cost efficiencies, scalability, business continuity and disaster recovery. This is a technology area which has really taken off over the last few years and technology advisors (Gartner / Forrester) are predicting big changes in the near future. These technology advisors say that cloud computing, to include just about every aspect of technology provisioning services, will be an extremely competitive service offering within the next 2-5 years. In fact, recent studies from both indicate that the private sector and public sector will move aggressively to Cloud Services over then next 2-5 years. Many big players are working feverishly to compete for this market space… (Amazon, Microsoft, Google, IBM, SalesForce.com, etc, etc.)
So, why would the state invest in a single, stand alone data center now? It seems that it will be out dated before it is complete. How can the state compete with the private sector for commodity services? And the ‘real cloud’ offers scalability and disaster recovery that a single state “private cloud” can’t offer. The ‘real cloud’ provides instances of customer data and services in multiple geographic locations throughout the country which provides real disaster recovery at a price cheaper than the state can ever offer. I’d bet $300 Million on it! (or is it $392 Million now…)
The NPR Reports:
DIS stating that they hadn’t really done a cost comparison with private sector offerings. What? And, this was funded?
DIS (Tony?) stated that the current building might not withstand another earthquake. Really? If the building is unsafe then why would the state let hundreds of people go to work in it every day? I assume there are no plans to demolish the building after the new data center is complete?
DIS (Jim?) stated that they are out of floor space and power in the existing space. What about server virtualization? All the data centers I’m aware of are gaining floor space and reducing power utilization simply by deploying virtualization technologies to consolidate servers and services as the hardware is due for life-cycle replacement.
I simply don’t get it. As a taxpayer, I’m embarrassed by this decision and can’t understand how something so poorly vetted could move through our legislative process so easily; especially now.
Thank you and good luck with your Bill.
-Hayden