Editorials

Do Big Data-Type Projects Succeed More Often?

It used to be said that IT projects (and since SQL Server is my focus, SQL Server database projects by inference), would fail an astronomically large percentage of the time.  This was because, in most cases, of scope changing so frequently, particularly during development in a “waterfall” style project approach that had an extended calendar and fixed endpoints for functionality and deliverables.  This was a huge piece of the push behind other development approaches (agile, for example) to allow for changes, while still delivering projects.

We did a survey a few years ago about big data projects and whether they were considered successes on completion.  The “failure” rate was considered to be more than 90%.  That failure came from not getting results that were expected or needed for the project.  The functional pieces might be wrapped up just fine, but the actual deliverable information and results didn’t meet expectations for the project.

Now, we’re seeing more projects where the deliverable is more about the information infrastructure and support for tools and decision-making processes but less about particular information or analysis completed to a specific end.  I sort of see it as if we were (finally) stomping our virtual foot and shouting out “FINE!  If you don’t know what you want, I’ll just give you the information and YOU figure it out!

Once the temper-tantrum is over, the success rate seems to be improving.  Now when we approach a project, the goal is to understand first HOW the information will be used, WHAT the information and data inputs will be, and then we determine how we’re going to provide for that type of information flow.  Making the raw information available in a manner that is helpful, can be massaged and is true, along with applying necessary protections, seems to be the key ingredient to succeeding with these types of information flow projects.

Do you see this with your projects as well?  If you look at what your deliverables are, do you see that you’re focusing more on providing data than you are fully-baked results or information?

Some examples of this –

The old way – provide reporting showing sales figures by quarter and geographic segment.
The new way – provide the data that we can use PowerBI or other tools to create reports.  We’ll need key elements like geographic segment, sales date and item numbers so we can create and manage the reports needed.

The old way – provide an inventory report showing sales by item
The new way – provide the data that shows our inventory and sales details for that inventory

In short, by focusing more on the data elements needed to provide the information, we get out of the analysis business (in these cases) and back into the data business.  Sure, there may be some combining of data to get to certain data elements, but the user is able to use their favorite tools to create reports, analyze the trends, etc.

We are indeed seeing happier resolutions to the projects, I’m curious how many other people in this area (data folks) are seeing that providing data, rather than data results, is something that is bringing in deadlines, and improving success rates on these key projects?