When it comes to building AI solutions, the recurring theme I observe, posed by decision-makers and business leaders in their questions, is: “How can we prevent AI projects from failing and ensure successful implementation? What is the key factor? How do we select an AI vendor? How can we avoid squandering a significant amount of resources and not realize it until 1-2 years later?”
So, what comes after setting the plan and goal? This is the planning and architecting phase, which is responsible for the failure of 85% of AI/ML projects post-implementation. This blog post will discuss these common issues and provide solutions.
Issue 1: Data Strategy
The primary obstacle that hinders the success of all AI projects is the absence of a robust data strategy, coupled with issues related to data quality and volume. This signifies that the organization is not yet primed for a successful AI transformation. It is imperative for them to invest in data or alternatively, construct a Proof of Concept (POC) to gain insights about their existing resources and future requirements, and then scale up if the POC proves successful.
However, there are instances where such data can be procured or readily available, or certain processes can be implemented to gather them, or they can be obtained through partnerships. If this is the situation, it’s a positive sign, and you can commence your project immediately. A common misconception is the assumption that having a large volume of data equates to readiness. However, a vast amount of data is futile if it lacks predictive quality or fails to provide actionable insights that yield valuable outcomes. This is why the lean methodology is effective. In this context, a POC will facilitate the learning process. If the POC is successful, it can serve as a foundation to build a production-level application.
Issue 2: Technical Expertise
Let’s assume that your data is well-managed, with appropriate data governance and infrastructure in place. The second most critical factor then becomes the lack of necessary skills. A successful AI project requires a diverse set of skills and solutions. These can be broadly categorized into Technical skills and Business skills.
In terms of technical skills, it’s crucial to have genuinely competent data science professionals on board. The key distinction between a data science/AI project and a traditional software development project lies in the inherent ambiguity and uncertainty of an AI project. In conventional software development, you might sequentially add features, building upon what you’ve accomplished up to that point. However, in AI, a change in approach could mean that almost everything needs to be redeveloped, and previous work might have to be discarded.
Identifying the right talent who can deliver results versus those who cannot is a challenging task during the interview process. Inexperienced or merely adequate ‘data scientists’ or ‘AI engineers’ often lead to false starts, flawed models that appear effective, and considerable wastage of time and resources. This is primarily because they may not be sufficiently familiar with the lifecycle of a data science project, which includes data engineering, data processing, building ML models, MLops, and developing and deploying production-level applications. Many data scientists in organizations tend to focus solely on model building, neglecting the other aspects. Model building constitutes only 20% of the entire process; it’s just the tip of the iceberg when it comes to real-life AI/data science applications. You need to have a data infrastructure in place, collect and label data, process it, and prepare the messy data for actionable insights. You need to build the AI model, deploy it to production, update it, maintain it, monitor it, manage MLops, and so on.
Hiring AI Professionals
When it comes to hiring AI Developers, Data Scientists, ML Engineers, and the like, my advice is to look beyond academic qualifications and degrees. The rapid pace of technological advancement means that knowledge can become obsolete within a year, or even months. Therefore, individuals who are quick to adapt and committed to continuous learning often outperform their peers significantly. The challenge lies in identifying and hiring these high-performers who can truly deliver. The consequences of a poor hiring decision may not become apparent until 9-12 months later, potentially resulting in substantial financial loss and wasted time. In most developed countries, the hiring process can take upwards of six months.
Moreover, there’s a risk that your recruitment process may not be optimal. If B-level employees are conducting interviews, they may end up hiring C-level candidates, who in turn hire D-level candidates, leading to a downward spiral in talent quality. Relying solely on a candidate’s degrees and past experiences can result in hiring an average performer who, despite impressive qualifications, may not be able to deliver effectively.
Therefore, it’s crucial to invest wisely in the hiring process and get experienced people to interview, or consider partnering with an AI firm that can provide the necessary expertise if you cannot afford an in-house data science team.”
Problem 3: Business skills
Issue 3: Business Acumen
The success of a Data Science team is not solely dependent on their technical prowess. A significant challenge lies in bridging the gap between understanding how AI technology can enhance business performance and customer experience (which requires business acumen), and the technical expertise needed to develop such solutions. This disconnect often results in a flawed product concept or roadmap, or an end product that fails to meet business requirements or is simply subpar and unusable.
Data scientists and AI professionals who possess both technical and business skills are rare and highly sought after. These individuals should be able to guide the tech team, stakeholders, and product people toward a common goal. They should be capable of explaining complex models to an audience unfamiliar with the intricacies of math and statistics, and link these explanations to the business benefits derived from the results. They should also be able to address stakeholders’ questions and concerns, fostering mutual consensus.
Without these skills, professionals may struggle to justify their work or explain it in an understandable manner, leaving stakeholders with partial knowledge and indecision, potentially leading to project paralysis. In some organizations, objections from a single key stakeholder could halt a project before it yields results.
Issue 4: Overcomplicated Projects
Enterprises often perceive AI projects as costly endeavors, leading to overly ambitious goals that promise transformative results and high returns on investment. However, these lofty objectives often result in unrealistic expectations and the failure of ‘moonshot’ projects.
At Aigenexpert, we advocate for the SMART goal approach, starting with a single, achievable solution to a specific problem. Achieving quantifiable results from this initial project can serve as a blueprint for AI adoption within your organization, paving the way for more successful AI initiatives.
Issue 5: Senior Management Hesitation
The final hurdle is the reluctance of senior management to invest or their tendency to withdraw prematurely. The high costs and risks associated with hiring Data Science and AI professionals, the lack of in-house expertise, and past experiences with companies that overpromise and underdeliver can foster a risk-averse attitude.
If you are a large organization with the resources to maintain a robust data science/AI team, or an AI-focused product company with a comfortable budget, you can build your own in-house team. However, it can be beneficial to occasionally engage with an external partner for consulting projects to gain additional perspectives and insights. For those with budget constraints or concerns about the high risks and costs of building a full-fledged data science/AI team, partnering with an external firm is a viable option. However, choosing the right partner is crucial. Opting for a cheaper vendor can be more costly in the long run if they fail to deliver. In Data Science and AI, the success of a project heavily depends on the skills and intelligence of a select few engineers, much more so than in traditional software engineering projects.
These are the five main challenges and my recommendations on how to navigate them. If you have a clear AI transformation roadmap and know what AI and data science solutions you need, and are looking for a reliable partner with expertise, integrity, and transparency, I strongly suggest you contact us today. I would be happy to be your thinking partner and ensure we bring the best resources to your aid. Looking forward to connecting with you soon!