Editorials

Big Data Projects Failure Rate?

I saw a recent post talking about the failure of Big Data Projects… and it made me wonder about the metrics we would use to judge success/failure.

I know that there are very well worn statistics about more traditional projects and the failure rate at companies when it come to waterfall methods and such. The statistics that I’ve seen are pretty staggeringly against the process and the expectations for projects. But I have to say, when I’ve asked about the companies that “failed” in their projects, the data can be interpreted pretty much however you like.

In some cases, the project is abandoned and just walked away from. I’d call this a failure – IF nothing was learned. Sometimes I’ve seen projects abandoned because something was learned about the target information. If the project exposed a fatal flaw – either early or in a place that wouldn’t have been otherwise known, is that a failure? But these complete “walk-away” type failures don’t seem to be the majority of failures in my experience.

Probably the most common traditional failure was the perpetual scope-creep solution. It just keeps going and going in terms of requirements and modifications and…

But with big data and IoT projects that I’ve seen posts about, it seems like people are suggesting that there are scant examples of big data solutions that have resulted in surprising or useful information.

I find that premise – that there are not successes – pretty stunning! And, frankly, impossible. We’ve seen many successful projects where the solution is based on “big data” that wasn’t used before and is available in eye-blurring detail – but was ingested and analyzed and made sense of with various tools and SQL Server. We’ve seen good payoffs in learning about what makes the transaction cycle tick, related items, etc.

Heck, Amazon.com’s “other’s have purchased” options during checkout are fine examples too of the types of things that can be learned from looking at related streams of data. That’s just a simple example, but…

What have you seen with your own projects? Do you think Big Data projects generally fail to produce actionable results? Have you deployed projects that provided information that is actively used in decision support for your (or your client’s) company? Inquriing minds want to know.