ANALYSIS ENGINES - A RANT

After attending a two-day Analysis IPR, or “in-process review” for those not familiar with the acronym, I’m annoyed and frustrated. The greatest analysis engine that has ever existed is the 3-lb lump of grey matter behind the eyes and between the ears, yet most attendees use it to drive their vocal chords instead.

I know I’m being over the top here; even a bit unfair, since the questions being asked aren’t easy to analyze, and the available data is chaotic, incomplete, and possibly even irrelevant. But polling (trend analysis in this case) doesn’t answer the questions being asked. It can’t; at its best, trend analysis can indicate correlation, but not causation.

What is needed is for the participants to quit using those analysis engines as vocal chord drivers and instead give some hard-headed thought to what effects can or should occur given the input changes being made. (That’s called hypothesis formulation, by the way.)

Then give an equal amount of thought to what measurements need to be made to either prove or disprove the hypothesis. That includes making sure that the measurements can in fact be made. I remember some years back a colleague of mine developed a brilliant analytic solution to a geolocation problem we had been working for years. His mathematical solution was complete, correct, and absolutely spot-on. But in order to do the mathematics, he had to structure the problem is such a way that the input estimation variables needed were not physically realizable. Meaning that the solution could never be tested, since the data needed to test his solution could never be measured. Back to the drawing board ....

Finally give another equal amount of thought about how to acquire the data. That means ignoring the mass of existing data until it can be shown to contain the relevant data. Then - and only then - begin to look at that mass of chaotic, incomplete, and possibly irrelevant data that’s been collected for every purpose except yours.

Here’s an analogy. The current analytic process is akin to taking three 1000-piece jigsaw puzzles, mixing them together, turning over every piece until only the backs show, putting them together, and only then turning the three completed puzzles over to see the pictures.

Did I mention that I’m annoyed and frustrated?