Revamped Statistics Policy: Big Data, AI, ML in Focus

Learn how to use big data, AI and ML to improve your official statistics with the new draft policy. Read on!

The Ministry of Statistics and Programme Implementation (MoSPI) has recently released the revised draft of its National Policy on Official Statistics (NPOS), which aims to provide a comprehensive and coherent framework for producing and disseminating official statistics in India. The draft policy highlights the importance of using big data analytics, artificial intelligence (AI), and machine learning (ML) to enhance official statistics' quality, timeliness, and relevance. It also emphasizes the need for collaboration and coordination among various stakeholders of the Indian official statistical system, including central and state governments, research institutions, the private sector, civil society, and international organizations.

The draft policy is a revision of the earlier draft released in 2018, based on the recommendations of the National Statistical Commission (NSC) and the United Nations Fundamental Principles of Official Statistics (UNFPOS). The UNFPOS, adopted by India in 2016, provides guidelines for ensuring official statistics' professional independence, impartiality, accountability, and transparency. The draft policy seeks to operationalize these principles in the Indian context and align them with the emerging data needs and challenges.

One of the key features of the draft policy is the recognition of the potential of big data sources, such as administrative records, satellite imagery, social media, mobile phone data, internet transactions, etc., to complement and supplement the traditional sources of official statistics, such as censuses, surveys, and registers. The draft policy states that the Indian official statistical system must reorient its strategies with emerging technologies, including AI/ML, to collect, compile, process, store, integrate, analyze, and disseminate data. It also acknowledges the need to address the issues of data quality, privacy, security, ethics, and governance in using big data for official statistics.

Another important aspect of the draft policy is creating an integrated data system (IDS), which will enable the sharing and linking data across different domains and levels of government. The IDS will facilitate the generation of core statistics, which are defined as a set of essential indicators that reflect the key aspects of national life, such as national income, production, services, budgetary transactions, money and banking, capital market, external sector, demography, social and environment sectors. The collection and dissemination of core statistics will be made mandatory for all levels of government and released periodically.

The draft policy also envisages innovation in survey design and methodology to reduce the burden on respondents and improve the response rate. It proposes to use mixed-mode data collection methods, such as web-based surveys, computer-assisted personal interviews (CAPI), computer-assisted telephone interviews (CATI), etc., to optimize the cost and quality of surveys. It also suggests using adaptive sampling techniques, such as responsive design and adaptive cluster sampling, to improve the efficiency and accuracy of surveys.

The draft policy further aims to enhance the quality of official statistics by adopting international standards and best practices. It proposes to develop a quality assurance framework (QAF) for official statistics, which will specify the quality dimensions, criteria, and indicators for assessing and improving the quality of statistical products and processes. It also plans to establish a quality audit mechanism (QAM) for official statistics, which will conduct independent audits and reviews of statistical activities and outputs.

The draft policy also stresses the timely dissemination of official statistics to meet the information needs of various users. It advocates adopting an advance release calendar (ARC) for official statistics, which will specify the dates and frequencies of releasing statistical reports and publications. It also recommends using various modes and platforms for disseminating official statistics, such as web portals, mobile applications, dashboards, infographics, etc., to enhance their accessibility and usability.

The draft policy is open for public comments till September 9th, 2023. The MoSPI invites feedback from all stakeholders on the draft policy through email at capcso-mospi@gov.in or aisha.saeed@gov.in. The final policy will be formulated after incorporating the suggestions received from various quarters. The NPOS is expected to provide a robust and modern framework for producing and disseminating official statistics in India that are reliable, timely, and credible.