In the previous blog post, I discussed some of the challenges facing investment committees responsible for analyzing, prioritizing and selecting specific data science budget requests. In this blog post, I discuss the challenges facing business stakeholders or analytics groups responsible for developing budget requests. For simplicity, we will call these individuals data science project owners.
Do we have everything?
In my experience, data science initiatives can be divided into three main phases. Phase one includes defining the requirements and gathering all the necessary data and artifacts. Phase two involves all the data engineering and model development. And phase three includes everything that’s required once a model is developed. In my experience, data science project owners do a good job planning and budgeting for phase one and phase two. It is phase three that we will discuss further.
One area of data science projects that is often not accounted for in the budget is the prototype phase. We have found that once a model is developed using and testing against historical data, the model should be deployed in a small, real-life scenario to determine the true efficacy of the model. This often involves working with the business stakeholders to design a controlled experiment in which a sample of decisions utilize the model and the results are compared against the status quo. Accounting for this phase, including the size of the experiment and the length of time required, is necessary to ensure proper budget and resources are available. This should be included in the initial budget request.
Getting to production
The second area to consider when developing a budget request is the process required to deploy a model to production. Depending on the maturity of your analytics platform, and the roles and responsibilities across analytics and IT, this process can vary significantly from organization to organization. Some organizations have a very efficient and seamless process for deploying data science solutions – including a development environment that closely mirrors the production area. Other organizations require more re-engineering of initial solutions in order to achieve the scale and reliability of a production solution. If this process is overlooked in the initial budget request, the project carries significant risk inactivation. Planning for this prior to submitting a budget request eliminates that risk.
Assuming a data science project goes according to plan – a successful model will be adopted by the business stakeholders. The next question to address is “Who maintains the model long term”. The degree of maintenance will depend significantly on the type of model deployed. Who maintains the model will also depend significantly on the analytics organization structure. Is there an established group dedicated to analytics OPS and maintenance? If not, then some degree of model maintenance should be articulated in the initial budget request. As analytics groups continue to build trust with business stakeholders, the proper level of maintenance and support for each solution is required. The initial budget request should consider funding for maintenance, as well as an estimate for training and supporting business stakeholders’ adoption of the model.
In many cases, the three areas I have discussed in this blog post are not required to develop and receive budget requests for data science projects. While not required, including these topics in your budget request presents a comprehensive assessment of the true effort required to not only develop a model, but ensure insights are activated and ROI is achieved.