Data Preparer – A market validation journey

You are currently viewing Data Preparer – A market validation journey

Data Preparer - A market validation journey

The Journey

In January 2019, our team received a grant under the iCURE programme for a market validation of our business proposition. The idea was simple: in a period of three months, we would get out of the lab to talk with as many potential customers as possible. The goal was to discuss their data preparation challenges, learn from them, and try to detect if there is a market fit for our data preparation solution.

In short, the assumption we set off to validate was that “there is a market for best-effort automated data preparation”. The motivating observation was, and still is, that data scientists spend most of their time preparing the data for analysis, rather than actually analysing the data. We would discuss our assumption with companies offering domain-agnostic business intelligence, analytics, and IT services, while also focusing on a number of vertical domains, including: e-commerce, financial services and market research.

We started off by fine-tuning our business model canvas and our verbal business card, and made a plan regarding the events we would visit and the people we would talk to. The goal was to have as many meaningful conversations as possible.  The schedule was loaded, including six events in three months, in Manchester, London, Leeds, Dusseldorf and San Francisco.

The Lessons

Overall, after tens of face-to-face discussions, it turned out that the vast majority of the people we talked to were very interested in data preparation, as it was among the most difficult problems they were facing. Automation was more than welcome, however:

  • The system would have to be as transparent as possible in explaining and justifying its decisions.
  • The system would have to allow steering of the automation. Potential users expressed the need for more control over the automation, so that the result would be as close to their requirements as possible.

Also, it was interesting to note that “self-service data prep” was recognised as a priority, and it was often promised, but rarely delivered. In demos during trade shows, typically we would see simple graphical frontends of data preparation software through which the user could drag and drop components and handcraft data processing pipelines. However, the level of configuration required to operate the software was without exception forbiddingly complicated to the less tech-savvy. This was most of the time contradictory to the core “self-service data prep” promise, and also in line with the feedback we received from a number of companies: developing in-house solutions was typically preferred over adopting one of the self-service solutions.

With user feedback in mind, we had a clearer view on the functionality our minimum viable product should encompass. We implemented greater transparency and steering capabilities, and integrated them into our current release. As such, the first version of Data Preparer, while it can still support complete automation, is now much more feature-rich than the prototype that served as the demonstrator during our discussions with interested parties.

As a conclusion, the market validation journey proved particularly valuable, for a number of reasons. First, we made a number of contacts in the domain. Second, we received feedback that (a) largely reaffirmed our initial assumption that there is a market need for automated data preparation, and (b) helped us prioritise feature development. Third, the whole experience ultimately helped us learn a lot of valuable lessons, informing our next steps.