Aritifical Intelligence in India: Opportunity and Challenges

In 2010, I hired two engineers from an Indian college to assist me develop a product that could automatically grade the spoken English ability of job applicants. About a year later, they knocked on my office door with concern etched across their brows: “We are doing machine learning here, but all our friends are doing software engineering,” they explained. “Do we have a future?”

Things have changed dramatically. In India today, “Artificial Intelligence” has become the new wave, and everyone wants in. Every engineer claims some type of machine learning project on her resume. The ubiquitous library management system, a software development project has now been replaced by a project to automatically recognizing handwriting. A cottage industry of training courses in AI, machine learning, and data science has blossomed throughout the country. Most businesses have a top-down mandate to incorporate AI in their processes and product. The excitement has reached all the way to the government: In this year’s speech on the federal budget, Indian Finance Minister Arun Jaitley announced that the country will launch a national program to promote AI research and development.

This newly-awakened interest in artificial intelligence borders almost on euphoria. However, India has some fundamental challenges. India has a relatively small body of researchers and research output in the field of machine learning. (Presently, AI is mostly machine learning).  The contribution of Indian researchers to top AI conferences constitutes one-fifteenth of the U.S. contribution and one-tenth of that of China. At the most recent AAAI conference, Indian researchers presented 23 papers compared to 381 presented by our US colleagues and the 237 that were from China. Most Indian research institutions have no or at best rudimentary AI research program. Indian Institute of Science, that has one of India’s most productive AI program contributed only 75 papers to the field last year, compared to the 700 that came from Stanford and the 500+ from Tshingua University. India contributes little to the creation of new knowledge in machine learning; but more worrisome is that we have little local expertise in the new knowledge that is being created every day by others. We do not have people who can train a new cohort of machine-learning engineers and scientists to develop and commercialize technology.  

India’s lack of research prowess is n  ot limited to artificial intelligence. In my recent book, ‘Leading Science and Technology: India Next?”, I survey the current state of scientific research in India. I calculate that we have only about one-seventh the number of productive researchers (garnering more than eight citations a year) as compared to China and a tiny one-seventeenth the number of the US. India’s largest technology universities can claim only about one-fifth the number of research faculty as American and Chinese universities, and they enjoy only about one-sixth their budget (in terms of PPP). Research is a lackluster career in India. The best students either go to the US or take up a career in the private sector in India. Further, the research community has limited exchange with the international community because of government and institutional restrictions on travel budgets. Overall, India doesn’t have a critical mass of productive researchers to progress technology and have an impact on our businesses.

India cannot afford to miss the AI revolution. The country needs to develop and commercialize AI if its businesses are to be globally competitive. AI could also help address the country’s human needs, impeded by corruption, lack of capacity and infrastructure. 

Likewise, the AI revolution needs India. The country’s diversity and complexity presents a rich set of challenging problems for artificial intelligence. Current AI techniques are limited in their ability to handle complexity, and they’ll have to mature to deal with the diversity of life in India. Also, the India forms the IT backbone of the world. The country’s companies and talent, are thus, the natural contenders to add ‘intelligence’ to all the digitization. Investment in India can help move the whole field ahead.

Impact on Indian Businesses

India’s global business activities have typically revolved around the IT services and business process outsourcing (BPO) industries. These businesses depend on India’s demographic dividend: the large population of English speakers trained in basic numbers, computers, and programming. This skilled workforce along with India’s various cost advantages have been powering growth in these service sectors for the last three decades.

The work of the BPO industry is comprised of several human intelligence tasks, such as transcribing speech, digitizing handwritten forms, and tagging images. With recent disruptions in machine learning — specifically deep learning — these tasks can be done by machines with a high degree of accuracy. Machine learning also adequately addresses tasks that require some basic analytical skills such as classifying documents, scoring them, and deriving structured data from them. A new AI-driven process requires humans only to check or correct low-fidelity predictions rather than to execute the task from scratch themselves. In another example, bot handles simple chat and email requests while directing more complex requests to human operators. Even in the latter case, machine learning helps generate possible response options for operators which they may select or modify. The call process is further away from deep automation, given the clunky automated call solutions.

Will the IT industry in India be similarly impacted by artificial intelligence? To get to the point directly, machine learning has not automated programming. The ability of AI to write a program for a problem statement is as primitive as its ability to understand natural language. The research community has made some progress in sub-problems (for example, my own company’s work in automatic grading of programs) but not much is of industry scale or been commercialized. Software engineering is becoming less human intensive due to better reuse through development of products, APIs and digital services. Automation is also helping to manage services beyond hardcore programming, such as network monitoring, testing, and infrastructure maintenance.

Despite the opportunities for the present and future, Indian companies have been slow to adopt AI. We see more deployment of ready-to-use technology such as speech recognition, face recognition, text analytics, and rudimentary chatbots. Many of these models are not trained for Indian data and do not work well. However, they are used anyway. India does not possess enough trained manpower to apply machine learning to our own problems and data, in spite of the number of standard packages available. I interview a lot of people who call themselves data scientists. This rather small community can code and run the supervised learning flow. They are unfamiliar with the basic concepts of machine learning and statistics, model selection, cross validation, bias and variance. A machine learning engineer needs to know these concepts in order to interpret results, and also to troubleshoot when the standard flow does not work. And the standard flow does not work quite often. This is where my small research group spends most of its time – staring at result tables and iterating over features, techniques and data sanitization to get a model that will work in practice.

Among India’s various industrial sectors, the BPO industry has most readily adopted AI innovations. The commoditized solutions of AI work well for them. India’s IT industry needs to prepare itself for offering AI services. A lot of digitization has already happened. The next big global demand will be for extracting and using intelligence from this data for business efficiency. The IT companies have started AI practices, and have conducted some training programs for their employees. Their biggest challenge is the lack of trained or experienced manpower in these areas.

For other sectors say, health, manufacturing, there is very little penetration. The country wishes to revive its manufacturing through a much publicized “Make In India” initiative. But there is little interest or capability in robotization — in contrast to China, which has made robotization a priority. Many companies slip on the first crucial step of identifying a business problem and converting it to a machine learning problem. They have a mandate to employ machine learning, but they do not know where or how to do so. A consulting company from South India wanted me to provide a reference for a company offering machine learning services, but did not know the business problem they wished to solve. Another company in finance wants to hire machine learning engineers, but do not know the skills they are looking for or how to interview candidates.

There is a lot of buzz in India around AI startups. In particular vogue are startups offering chatbots for various use cases. Unfortunately, I am yet to see a company that has developed a unique innovative application of AI or developed a disruptive new technology. Our technology and internet companies rarely publish in AI conferences. Google, Facebook, LinkedIn, Baidu, Alibaba, and Tencents are big contributors to AI conferences, software libraries, and open source softwares. Private labs in India publish as well, but these are typically Western entities such as Microsoft Research, IBM Research and Xerox Research.

Notable exceptions include TCS Innovation Labs, Strand Life Sciences, Aspiring Minds (my own company). Strand Life Sciences, co-founded by an MIT alumnus, works in specialized diagnostic methods and precision medicine. Aspiring Minds uses AI to make the recruitment process efficient and meritocratic. We have published on automating assessments of programming, spoken English, soft skills and motor skills. I hire candidates with Bachelor degrees who could make through the selection process of the top graduate programs in US. They have great programming skills and understand how to approach open-ended problems. I train and mentor them in applied statistics and solving real-world machine learning problems. One such person had left a Microsoft offer to join us, and is today at MIT after a five years stint with us.

There is understandable concern in India for how machine learning and automation might impact jobs. The BPO industry did not register any decrease in hiring last year. We can assume that any losses due to the automation of processes in one area were counterbalanced by the growth in volume in the process or the creation of new ones in other areas. The IT industry, on the other hand, hired in much smaller numbers last years, than in the years before. While we may attribute some of this decline to automation, it is more likely a fact of the IT industry’s usual business cycle. We can note at least two similar hiring slowdowns in the last ten years, usually coinciding with a US or global slowdown or recession. The job outlook is projected to be more positive this year. This is nothing to cheer. These industries are awaiting a massive disruption in the next five to ten years. We need to be readying ourselves for the future by building skills in those areas that will not soon be automated. It is time for India to up its game.

What should India do?

India has to adopt AI in a big way to sustain its journey of socio-economic upliftment of its people. For the last 30 years, the services industry has driven the strong economic growth that has been the vehicle for India’s development. Those industries were undergirded by India’s educated English-speaking population in combination with India’s traditional cost advantages. But in this second machine age, massive population is not a competitive advantage. Indian service companies will be competing with international companies that augment human workers — or even replace them completely — with sophisticated AI algorithms. Furthermore, the world will be looking to India to provide data science services in much the same way that India has provided IT and IT-enabled services in the past.

The first step in doing so is to attract a critical mass of AI experts: individuals who hold PhDs from world class universities. Unfortunately, India is not a choice destination for top researchers. Even among Indian PhD students who study in the US, only about 8.7% express any desire to pursue a career in India. They are of the opinion (correctly) that research careers and faculty careers in India do not pay sufficient dividends in either the personal or professional sense. In a recent white paper, I detail a plan on how the government can help assemble a team of 500 AI researchers in India’s public institutions over the next five years by instituting an attractive AI fellowship program for faculty and PhD students. In parallel, we need private initiatives- individuals setting research universities and research institutes in India. A hundred AI experts working on developing and disruptive technology under a single roof in India could provide the catalyst and momentum that India needs to spark a movement.

Together with researchers, India needs to develop a proper ecosystem to fully realize the impact of emerging technology on India’s social needs and industrial progress. India has huge issues of capacity, access and bureaucratic challenges of speed and accuracy in areas of healthcare, banking, sanitation, agriculture and education. There are also huge corruption issues given the human element in various services. Also, India’s diversity of language, scripts, dialects, accents, dress, and culture presents a rich set of challenging problems for artificial intelligence. It will demand the current AI techniques, which are too sensitive to the training data, to mature to handle complexity and diversity. The mismatch between the scale (population) and the lack of capacity in India also poses different and challenging problems. For instance, where researchers in the US hope AI can make doctors more efficient, in India the question is how AI can do the job of a doctor in rural areas that currently have no medical care at all.

Artificial Intelligence gives India the opportunity to leapfrog some of its issues, be they automatic processing of applications, cheap diagnostic methods, or learning/teaching aids. For example, one young entrepreneur from Rajasthan showed me a system that can analyze images of certain grains to ascertain their quality and estimate the price they will likely fetch at market. A system such as this can help level the playing field between farmers and wholesale buyers. Another example is an automated teaching assistant for programming skills, a project my team is currently working on.

India needs to spur the research community and industry to work on such problems. India’s Department of Science and Technology could hire program managers to frame hypotheses around problems that could be solved using AI, and then fund research programs for these problems through grants. Researchers could bid for these grants in open competition by devising a variety of approaches and solutions to the research problem. This would help create large useful labeled data sets and the technology needed for India. Students who work on these projects will naturally go on to create startups around them. Government should do impact evaluation for the technologies created and select worth ones for implementation. India’s space and defense research organizations could enact similar programs involving the AI research community to develop solutions for them.

The last and important piece is around creating impact on industry. The first direct impact is providing trained manpower to the industry. The AI researchers will be great teachers.  There will be a need for all levels of preparation programs: doctoral, masters, professional certification courses, one-off boot camps, company in-house training, etc. India could avail itself of the distinction of being the first to institute an AI curriculum in K-12 and introduce the subject in first year of engineering training. My colleagues and I performed a small but highly successful experiment (probably the first of its kind) teaching a data science class for grades 5-8 (supervised learning with unstructured data;

Training is great, but not enough. There is a need for a greater collaboration between the industry and research community. In their symbiotic relationship, industry provides problems and data to the research community while the research community develops algorithms and solutions. Finally, industry creates scalable software libraries for the algorithms for large scale usage and impact. To facilitate such an exchange, the government can create nodal centers across India to be run by universities. These nodal centers will run outreach programs to industry, house funds for joint industry-academia projects, and maintain tools and libraries.

The Indian industry needs to respond, if they wish to continue to exist! Our companies helped drive India’s progress over the last three decades by creating capacity for basic programmers and supporting undergraduate programs in the universities and institutes. They need to shift gears – start building teams comprising of PhD degree holders and aid university PhD programs to deliver quality. India’s future relies on developing manpower that can make artificial intelligence do interesting things and do what artificial intelligence cannot do by itself.

(A shorter version of this essay was published by MIT Technology Review here)

May 2018