An EDGE - ValueNotes Report on:

Examinations and the Role of Technology: Emerging Directions

An initiative by

Supported by

March 2010

Table of Contents
1.Indian Higher Education: Need for Exam Reforms........................................................................7 2.Summary Findings.......................................................................................................................... 10 2.1 Growing global higher education exam market faces many challenges...................................10 2.2 Online assessment has manifold benefits and addresses many challenges............................11 2.3 Indian Higher Education: On road to change and a long way to go.........................................12 2.4 Technology in exams: The Indian higher education context.....................................................13 2.4.1 Many potential rewards for Indian educational institutions..............................................13 2.4.2 Infrastructure the main challenge for Indian higher education.........................................14 2.4.3 Online exams in India - Future plans...............................................................................15 2.4.4 The opportunity is huge....................................................................................................15 2.5 Indian students ready for online exams....................................................................................16 2.5.1 High awareness and readiness but low experience with online exams...........................16 2.5.2 Perceived / expected benefits and issues........................................................................17 2.6 Conclusion................................................................................................................................ 18 3.Importance of Assessment in Exam Management.......................................................................19 3.1 Introduction............................................................................................................................... 19 3.2 Typical exam process...............................................................................................................19 3.3 Importance of measuring outcomes ........................................................................................20 3.4 Types of exams and assessment tools.....................................................................................21 3.5 Trends and innovations in university assessment....................................................................22 4.Global Trends in Online Assessment............................................................................................24 4.1 Online assessment: Increasing scope......................................................................................24 4.2 Online assessment: Increasing acceptance.............................................................................25 4.3 Use of technology in assessment set to grow further...............................................................26 4.4 Diverse applications of online assessment ..............................................................................27 4.5 Customized as per regional needs...........................................................................................27 4.6 Examples of online exams........................................................................................................28 4.6.1 Utah Education Network .................................................................................................28 4.6.2 Graduate Record Examinations (GRE)............................................................................28 4.6.3 Test of English as a Foreign Language (TOEFL).............................................................28 4.6.4 Graduate Management Admission Test (GMAT)..............................................................28 5.Indian Higher Education: An Overview.........................................................................................29 5.1 Indian higher education: A snapshot.........................................................................................29 5.1.1 Budget allocation and plans Big but inadequate...........................................................30 5.1.2 A huge demand-supply gap..............................................................................................31 5.1.3 Higher education: Quality issues......................................................................................31 5.2 Going forward: Indian education needs quantity and quality....................................................32 5.2.1 Driving changes at the policy level...................................................................................32 5.2.1.1New bills and regulations............................................................................................32 5.2.1.2Increasing competition................................................................................................32 5.2.1.3Need for consistency and deregulation.......................................................................33 5.2.2 Demand will drive growth.................................................................................................33 5.2.2.1Rising middle class driving demand............................................................................33 5.2.2.2Interest in the sector rising..........................................................................................34

5.2.3 Gender parity and greater awareness drives growth.......................................................34 5.3 Indian higher education: long way to go...................................................................................34 5.3.1 Planning and policy makeover.........................................................................................34 5.3.2 Higher education needs the ICT push..............................................................................35 5.3.3 Survival of the best...........................................................................................................35 5.4 The Indian exam market...........................................................................................................35 5.5 Challenges in traditional exam management............................................................................37 5.5.1 Administrative challenges.................................................................................................37 5.5.2 Exam design challenges..................................................................................................39 5.5.3 Resource allocation challenges.......................................................................................39 5.6 Tackling exam challenges through technology ........................................................................39 5.6.1 Increase in experiments to increase assessment effectiveness......................................40 Gujarat Technological University takes entrance exams online................................................40 5.6.2 Managing with less resources..........................................................................................41 Case study: NMIMS move to online exam ...............................................................................41 5.6.3 Rising importance to security and controlling malpractices.............................................41 Case study: Manipal University - One of the front-runners in online exams.............................41 Case study: University of Pune looking for better practices......................................................42 6.Technology in Indian Higher Education Exams............................................................................43 6.1 Online examination: Intent........................................................................................................43 6.1.1 High interest in conducting exams online........................................................................43 6.1.2 Future plans.....................................................................................................................44 6.1.3 Lack of clarity................................................................................................................... 45 6.2 Online exams: perceived benefits.............................................................................................45 6.2.1 Benefits: Lower costs.......................................................................................................46 6.2.2 Benefits: Increase in reach...............................................................................................46 6.3 Online exams Key concerns..................................................................................................46 6.3.1 State-of-the-art infrastructure...........................................................................................47 6.3.2 Resistance to change.......................................................................................................47 6.3.3 Adaptation of exam format...............................................................................................48 6.3.4 Unsure about the value of investment..............................................................................49 6.4 Online exams: Execution..........................................................................................................50 6.4.1 Execution: Automated administrative tasks......................................................................50 6.4.2 Execution: Establish an efficient system..........................................................................51 6.4.3 Execution: Establish a credible system............................................................................52 6.4.4 Execution: User friendly systems.....................................................................................52 6.5 Conclusion................................................................................................................................ 53 7.Student Speak.................................................................................................................................. 55 7.1 Profile of respondents...............................................................................................................55 7.1.1 General information..........................................................................................................55 7.1.2 Ownership of technology..................................................................................................57 7.1.3 Academic use of technology............................................................................................58 7.2 Awareness and experience with online exams.........................................................................59 7.2.1 Most students are aware of online exams........................................................................59 7.2.2 Majority of students have no experience with online exams............................................60 7.3 Readiness towards online exams.............................................................................................62 7.3.1 Conducting online exams sends positive messages to students.....................................62

7.3.2 Online exams score high on security and credibility........................................................63 7.3.3 Students feel online exams are far more accurate..........................................................64 7.3.4 Students have preferences in type of exam to take online..............................................66 7.3.5 Students more likely to join a university which conducts online exams...........................67 7.4 Perceived benefits and key concerns.......................................................................................68 7.4.1 Students see many important benefits of online exams..................................................68 7.4.2 Support and difficulty of usage the key concerns.............................................................69 7.5 Conclusion................................................................................................................................ 69 8.Conclusion....................................................................................................................................... 71 9.About ValueNotes............................................................................................................................ 73 10.About EDGE Forum.......................................................................................................................75 11.About MeritTrac............................................................................................................................. 76 12.About Excelsoft............................................................................................................................. 77

Table of Exhibits
Exhibit 1: Challenges in university exam management........................................................................11 Exhibit 2: Megatrends in higher education and their impact on assessment........................................12 Exhibit 3: Online examinations many potential rewards....................................................................14 Exhibit 4: Challenges to online exams..................................................................................................15 Exhibit 5: Are online exams a part of Indian higher education's plan?.................................................16 Exhibit 6: Perceived / expected benefits of online exams....................................................................18 Exhibit 7: Perceived / expected issues with online exams...................................................................19 Exhibit 8: Typical exam process...........................................................................................................21 Exhibit 9: Interrelatedness of assessments to course curriculum objectives........................................22 Exhibit 10: Types of assessment tools..................................................................................................23 Exhibit 11: Megatrends in higher education and their impact on assessment......................................24 Exhibit 12: Online assessment process ...............................................................................................26 Exhibit 13: Online assessment Questions and tasks.........................................................................27 Exhibit 14: Structure and size Indian higher education .....................................................................30 Exhibit 15: Budget allocation for higher education ..............................................................................31 Exhibit 16: Comparative Gross Enrollment Ratio.................................................................................32 Exhibit 17: Market size of Indian higher education exam market.........................................................37 Exhibit 18: Challenges in university exam management .....................................................................38 Exhibit 19: Interest levels in online exams are high .............................................................................45 Exhibit 20: Moving towards online exams?..........................................................................................45 Exhibit 21: Lack of clarity on implementation time frame.....................................................................46 Exhibit 22: Online examinations many potential rewards..................................................................47 Exhibit 23: Infrastructure a major impediment to online exams............................................................48 Exhibit 24: Major reluctance to increasing budgets for online exams...................................................50 Exhibit 25: Reluctance to increase exam fees for introducing online exams........................................51 Exhibit 26: Increasing use of technology in exam process ..................................................................52 Exhibit 27: Preference of time frame for multiple batches....................................................................54 Exhibit 28: Age brackets and gender distribution of respondents.........................................................56 Exhibit 29: Distribution of respondents' academic level and city of study.............................................57 Exhibit 30: Distribution of respondents' stream of study.......................................................................57 Exhibit 31: Ownership of computer/laptop and Internet connection.....................................................58 Exhibit 32: High frequency of usage of technology..............................................................................58 Exhibit 33: Similar academic use of technology across city tiers.........................................................59 Exhibit 34: Most students are aware of online exams..........................................................................60 Exhibit 35: Students originating from Tier III cities are less aware of online exams.............................60 Exhibit 36: Students who are not aware of online exams do not possess an Internet connection.......61 Exhibit 37: Majority of students have no experience with online exams...............................................61 Exhibit 38: Tier I students have more experience with online exams...................................................62 Exhibit 39: Conducting online exams sends positive messages to students........................................63 Exhibit 40: Students originating from Tier III cities far more impressed with online exams..................63 Exhibit 41: Online exams score high on security and credibility...........................................................64 Exhibit 42: Experienced students perceive no difference in security....................................................65 Exhibit 43: Students feel online exams are far more accurate.............................................................65 Exhibit 44: Speculation among students who haven't given online exams..........................................66

Exhibit 45: Students who feel accuracy is higher are more likely to join a university which conducts online exams........................................................................................................................................ 67 Exhibit 46: Students have preferences in type of exam to take online.................................................67 Exhibit 47: Students more likely to join a university which conducts online exams..............................68 Exhibit 48: Perceived / expected benefits of online exams..................................................................69 Exhibit 49: Perceived / expected issues with online exams.................................................................70 Annexure 1: List of participating universities and exam boards...........................................................79 Annexure 2: Delivering online exams consistently...............................................................................80 Annexure 3: CAT in mousetrap: A failure of management, says expert ..............................................82

1.

Indian Higher Education: Need for Exam Reforms

The Indian higher education sector is in the midst of tremendous growth and change. The population demographic has created an unprecedented number of young people in this country. The scale of the problem that India faces in terms of educating and providing employment to her vast youth population, is unprecedented. No country has had to educate several million people in a twenty-year period! The sheer size of the problem has made education a subject of national debate, with the attendant glare of the media and public opinion. In the past year, giving further impetus to the debate is the political push by the Human Resource Development (HRD) minister, Mr. Kapil Sibal and his views on the need for reform in the Indian education system. Not only do we not educate enough of our population, but many graduates cannot find employment. Indian and multinational employers complain about the 'unemployability' of the Indian student population. Some surveys suggest that as little as 25% of technical graduates and 10-15% of other graduates are, in fact, employable on course completion. Private diplomas and training by employers can only do so much. The issue must be tackled head-on at the origin - the learning being imparted and assessed at the university level. In the last few years, much debate has taken place with respect to reforming course curriculum and learning pedagogy. There is a greater emphasis on raising current standards to make the curriculum more relevant to professional requirements, and in line with international norms. The growth of private sector educational institutions and universities has helped improve supply, in terms of the number of seats available. However, these have also worsened the quality problem. Not only is there wide variance in the quality of graduates from different institutions, there are too many with unacceptably low standards both in the government and private sectors. In the quest for quality, even as efforts are made to enhance student learning by improving curriculum and pedagogy, or by easing entry for the private sector - there remains a critical element that is still missing from the picture the measurement of learning outcomes. Intended learning outcomes are the essential elements of a taught course, that are tested by universities through exams held at various levels:

Entrance exams test potential candidates on minimum acceptable standards for entry to courses. In-course (internal) exams help students understand the need for further guidance, improvement and feedback.

End-course (final) exams are final verdicts of student performance, to determine the knowledge, skills and abilities imbibed by the student from the course.

According to our calculations, the number of examinations in the higher education segment in the country, for just the Entrance and End-Course exams, are well over 200 million today. With steadily rising enrollment numbers and new courses being introduced each year, this number is expected to grow substantially in the coming years.

Indian educational institutes spend much time, money and effort to manage these exams, which are as yet being conducted through traditional paper and pen methods with manual evaluations. The prevailing system is riddled with many challenges, including:

Question paper leakages Student cheating/impersonation Shortage of qualified examiners Delays in paper correction Discrepancies in handling of transcripts Manipulation of scores Delays in results processing and declaration Handling of re-evaluations

In the last 5 years, there have been instances of universities that have used technology to overcome some of these issues: BITSAT by BITS Pilani, Manipal Universitys UGET, GCET Online by Gujarat Technological University, CAT by the IIMs, IIIT Bangalores Online Entrance exams, Delhi University online exams, etc. Many of these pioneers are even looking at how to change the model of the exam itself, thereby eliminating several of the existing set of problems. A lot of these exams have been successful. Clearly technology seems to be the answer to the existing issues. Granted, the adoption as of today is very low, but the argument remains; if these institutions were able to realise significant benefits through the use of technology in their exam process, it is definitely worth replicating the successes in the rest of the sector. Seeing the inherent benefits, there are many Indian exam bodies, such as AIEEE and GATE engineering exams which are considering a migration to the online format. The CBSE board has also made several recommendations involving exam reforms, many of which center around the adoption of technology. Internationally too, there is strong evidence that supports the growing use of technology in exam management. Large scale internationally recognized exams such GRE, GMAT and TOEFL have begun holding their examinations online, and many universities such as Washington State University, Cambridge and others are also experimenting with how to best leverage these new technologies. The US and UK are especially ahead of the learning curve, with technological solutions being sought and developed for all types of exams. Applications are being developed that allow mathematics and engineering students to derive and design online through authoring solutions. In sharp contrast, mostly objective type, multiplechoice based questions have been preferred for the online format in India. In India, though the number of exams that leverage technology are currently insignificant, the questions we faced were: Is this the beginning of a trend? Can technology really make a difference? There was no structured study in India at all to understand key stakeholders' (viz: exam authorities and students) thoughts on online exams. What are the perceived benefits? What are the key concerns? How ready are we as a nation to embrace technology and drive exam reforms in order to educate our millions of children?

This report embarks upon a journey to understand these issues through a first-of-it-kind comprehensive study based on extensive primary research. This study by ValueNotes, in collaboration with the EDGE Forum is an initiative by MeritTrac and seeks to understand the readiness of key stakeholders in Indian higher education to embrace technology in exam processes, their perceptions on how technology can help improve learning outcomes, and their concerns. We also hope that the report helps enhance the awareness amongst decision makers on best practices and the likely benefits and challenges. ValueNotes interviewed 75 education experts and university officials, from universities and exam boards across India (refer pg. 44 for distribution). The list of participating institutions is appended to the report (refer to Annexure 1, pg. 79). The questions we asked exam authorities included such themes as:

Details of current exam management process Interest and experience with online exams Expected / perceived challenges with implementing online exams Expected / perceived benefits of online exams that could influence adoption Future plans for technology in general, and online exams in specific Views on adoption trends at the national level

To understand the views of students with respect to online exams, ValueNotes also interviewed 400 students across India, with equal representation in Tier I, Tier II and Tier III cities. Questions revolved around the themes of awareness, readiness and experience with giving competitive and university exams online. Key questions included:

Awareness of exams being conducted online Perceptions of educational institutions which conduct exams online Perceptions of security and accuracy of online exams Willingness to experiment and confidence in the system Expected / perceived benefits and challenges for online competitive and university exams

This report captures the outcome of this survey and is structured on the following lines: 1) Types of exams and assessment tools, and a discussion on the typical exam process. Trends in global assessment are also discussed, with an introduction to online assessment. 2) The growth of online assessment across the world with some examples of successful implementation. 3) Snapshot of Indian higher education segment, featuring size, growth, growth drivers, key trends, success stories and challenges. 4) 5) The mind of the exam authorities: Thoughts on the future of online exams in India The mind of the students: Are they ready to migrate to the online format?

10

2.
2.1

Summary Findings

Growing global higher education exam market faces many challenges

The global higher education exam market, as discussed in Chapter 3, is huge, and growing every year. The number of exams held annually can be estimated at well over a billion, though variances in exam patterns across universities and countries could make this a conservative estimate. Exam management is a vital, yet a challenging part of university functions. The challenges in exam management can be classified in the following buckets:

Exhibit 1: Challenges in university exam management

Challenges in university exam management

Administrative challenges Registration Security and impersonation Valuation and scoring Declaration of results Exam logistics

Exam design challenges Level of difficulty Importance of internal and informal testing Designing for mass testing

Resource allocation challenges Budgetary constraints Availability of qualified examiners

Source: ValueNotes Research

Assessment, an essential part of university exam management, for the purpose of this study can be defined as, planning, conducting, grading and evaluating student examinations. Trends in global higher education segment have also influenced innovations and trends in university assessment. The following table enumerates these global trends and their impact on assessment:

11

Exhibit 2: Megatrends in higher education and their impact on assessment

Megatrends in global higher education Globalization Internationalization Impact on assessment Reason Growing cross border education models Growing participation of employers Mass assessments through technology Innovations in assessment tools New teaching and assessment techniques are emerging to accommodate off campus learning. Employers keen on closing the skills-gap for fresh recruits are getting more involved in student assessment at the university level. With more students giving entrance exams every year, institutions have been forced to re-look at assessment methods to facilitate mass testing. Competitive universities are trying to differentiate the learning experience by innovations in assessment techniques such as specialized knowledge measures, rubrics, portfolios, external judgments of performance, performance evaluation other than grades, student interviews, and employer surveys. While ICT has also been instrumental in aspects of curriculum delivery and other administrative functions, its role is vital in driving improvements in student assessment.

The massification of higher education Student experience has become more complex, with more options, opportunities and challenges for universities to address The transformative role of Information and Communication Technologies (ICT)

The growth of ICT in assessment

Source: ValueNotes Research

2.2

Online assessment has manifold benefits and addresses many challenges

Chapter 4 further discusses the scope of this study 'e-assessment' also known as 'online assessment'. Online assessment is a step towards making assessment more secure, transparent and fast. As education moves to becoming more learner-centric, assessment techniques need to keep pace with this evolution. The global trends in online assessments, recaptured below, reveal future trends in adoption:

Use of technology in education and assessment set to grow - Online assessment has become easier with growth in computer and Internet penetration. Technology has also improved with faster and more accurate computers and applications.

Diverse applications of online assessment - Online assessment is currently used more for formative tests, which include online quizzes and discussion forums. Summative online assessments are taking a little longer to move online, due to issues such as security, authentication and access involved. The greatest impact of online assessment has been on the administrative tasks. With almost 50% of the assessment cycle consisting of administrative tasks, these should be the biggest and fastest portion to offload. Distance and open education are likely to adopt online assessment fastest. However, for evaluation of descriptive answer sheets, a technology solution called On-Screen Marking has been revolutionizing the world of assessments. This technology allows scanning and anonymizing answer scripts, and then sending it across to scores of evaluators across locations. These evaluators would then assess these answers online on the screen and the scores are collated at the back-end.

Online assessment augments flexibility - Universities may suffer disruptions affecting course and exam schedules. This increases their need to automate systems. Industry innovators are struggling to keep up to the demands of universities. Secure, user friendly modules and easy to install solutions are the need of the hour.

12

Customization of assessment The prevailing scenario in developed and developing countries is very different. Universities in countries like the United Kingdom, United States, Canada and Australia are way ahead of the curve, in experimentation and implementation, with even essay type questions and diagrams going online. Universities in the developing parts of the world are still struggling with access to knowledge, infrastructure and acceptance. The primary driver to adopt online assessment may vary depending upon the requirements of a region.

Online assessments offer manifold benefits to educational institutes, and address many challenges faced in conventional exam management. The online exam market has been growing in the past few years, as more universities experiment with various formats, according to regional/structural needs. From the trends discussed, it is evident that the market is poised to grow significantly in the next few years, as greater technological advancements are made in the field.

2.3

Indian Higher Education: On road to change and a long way to go

Indian higher education has received a lot of attention in the last year, as pressure from public and media mounts for reforms. The Education Minister, Mr. Kapil Sibal has also reacted positively by announcing a lot of changes. As the stakeholders, viz. , universities, exam boards, educators and students get used to this new found status, it is now completely up to them to utilize this opportunity and bring about the long sought changes. The focus areas identified for changes include:

Changes at policy level: Three bills are due to come up during the budget session of the parliament. There is greater importance being given to regulation and improvement in the quality of education. Discussions on allowing private and foreign players are increasing, and their entry has the ability to change the scenario of the education sector completely, making it more competitive, result oriented and quality conscious.

Deregulation and consistency: We need long term policies, and non-governmental bodies to look after the needs of the education sector. With each new government and minister, new announcements are made, which lead to major upheavals. Organizations that work for the betterment of education sector can ensure consistency and thus continuity of efforts. Institutions and universities need more powers. Deregulation will allow them to experiment in a fiercely competitive environment.

Demand will ensure growth: The Indian middle class is the new market driver. The demand for good quality education has built pressure. Similarly there is greater acceptance of the idea that education is the answer to poverty. Banks, investors and venture capitalists have also identified the education sector as the next big investment opportunity.

13

Technology needs further improvement: ICT has been identified as an important tool to encourage education for the masses. However infrastructure and quality are big concerns. The Indian education sectors lags other sectors in adoption of technology. The UGC drive to encourage its use will help universities and educational institutes to attain the basic standards of technology to improve delivery and implementation.

2.4

Technology in exams: The Indian higher education context

Our survey, where we talked to decision makers in universities, national institutes of importance and other experts, brought out the fact that all of them see technology as an important tool to bring about changes in the Indian education sector. Issues such as greater transparency, increased geographical reach, improving quality of education, and others can be dealt with more successfully with introduction of ICT into education. This chapter covers in detail the benefits and challenges as perceived by our respondents.

2.4.1

Many potential rewards for Indian educational institutions

Following are some of the top benefits that may influence the decisions of the respondents:

Processes and assessment set to change: With the advent of technology into exams, the lengthy processes are expected to be faster, more efficient and fair. Assessment is also going through many innovations and is moving towards becoming a more robust system, where all stakeholders are aware of student performance.

Exhibit 3: Online examinations many potential rewards

Faster results Greater transparency Technology led student authentication Uniform assessment Higher security Manpow er savings Paper free, greener process Shorter exam cycle Overall low er cost Increased geographical reach Perception of progressive university (by students) Convenience for students (batches) 0% 20% 40% 60% 80% 100%

Very important

Somewhat important

Less important

Source: ValueNotes Research

14

Greater importance to security of exams: All universities expressed concern about cheating and other malpractices in exams. They use various methods to curtail it, and hope that technology will help them in this endeavour.

Automation of administrative tasks To reduce pressure on the university administration and staff, a number of universities are planning to increase automation and reduce administrative and repetitive tasks. This will also help them cut costs and lower manpower requirements. Automation of processes and availability of facilities online will make it easier for students, who want to appear for multiple entrance exams or attempt university and entrance exams during the same duration, as they do not have to stand in queues and travel to multiple locations.

2.4.2

Infrastructure the main challenge for Indian higher education

Following are some of the top challenges that may influence the decisions of the respondents:

Exhibit 4: Challenges to online exams

Infrastructure Lack of suitability (format) Non-feasibility of multiple batches Technical expertise Budget restrictions Faculty/staff apprehensions Security / reliability Authority Students' resistance Vendor information / selection 0% 20% 40% 60% 80% 100%

Major challenge

Somewhat challenging

Not a challenge

Source: ValueNotes Research

Resistance to change Many cited student and teacher resistance as a challenge and that since their current processes were working fine they would prefer to retain that system. Since not much was known about creation of online exams, format and adaptability also came up as concerns. Majority of the respondents expressed opinion that the current format of exams allowed them more choices and they did not believe that online assessment would be able to handle it.

15

Lack of infrastructure Online assessment will require a good and wide spread infrastructure complete with computers, Internet, and support from technology companies, and India currently does not possess sufficient resources to move. Many feel that awareness about computers is low among the rural students.

Lack of knowledge Size of universities and therefore ease of implementation, lack of knowledge on best practices and hard data were other major issues that came up during the survey.

2.4.3

Online exams in India - Future plans

The majority of exam authorities (68%) said they want to implement online exams in the future. This calls for a lot of effort on the part of the entire structure that will help in the transition. Going forward, Indian higher education needs to be equipped with trained teachers, infrastructure, technology that can adapt to Indian requirements, which in turn calls for a lot of experimentation, innovation in assessment techniques, technology upgrades and readiness to adopt changes and failures.

Exhibit 5: Are online exams a part of Indian higher education's plan?

9% 2%

Plan to increase / introduce online assessments Do not plan to increase / introduce online assessments Considering increasing / introducing online assessments Can't foresee

21%

68%

Source: ValueNotes Research

With almost half the respondents believing that less than 10% exams will be online in the next 2 years, and nearly 50% looking to go online in the next two years, the gap or opportunity is tremendous. It establishes the need to share knowledge, experience and beliefs about the entire process of going online. It also firmly establishes that improvement is assessment using technology is the way to go.

2.4.4

The opportunity is huge

The Indian Higher education segment has nearly 14 million students. Taking into consideration the average number of exams and the cost of exams we estimate the university exam market to be in the range of Rs 4,100 6,500 million. We believe this is a conservative estimate, as internal exams and other aspects are not included in the estimate. Similarly, based on estimates of the number of exams and the students who

16

appear for these entrance exams, we believe the value of the entrance exam market is in the range of Rs. 31,200- 45,900 million. No doubt a conservative estimate, as even for the higher figure, we have not considered certification exams such as CA, CS, and others. Further, more and more applicants appear for Indian higher education entrance exams every year. As an example, the CAT exam (taken by 250,000 students in 2009) witnesses an increase of nearly 40,000 candidates every year. Accordingly, it can be expected that the market will grow by about 10% per year, as costs are rising faster. These figures certainly justify the need for a step up in efficiency in many areas of exam management. Automation and the use of technology in exam management could save universities and exam boards around 20-50% of their overall costs, depending on the level of implementation. Given the huge size of the market and the clear cost and efficiency benefits, we believe penetration will increase dramatically over the next five years. However, with rising expenditures on one side, and cost and efficiency gains from technology on the other, it is safe to say that technology will play a major role in the Indian higher education exam market in coming years.

2.5

Indian students ready for online exams

During our conversations with university decision makers, many mentioned that one of the reasons behind the slow growth of online assessments in the country is the hesitation on the part of students. However, the exercise of taking exams online will yield many benefits to students. To bridge the perception gap between university officials and students, ValueNotes conducted a nationwide survey of 400 students from across Tier I, Tier II and Tier III cities, studying mostly at graduate and postgraduate levels. The aim was to capture the students' awareness, readiness and experiences with online examinations.

2.5.1

High awareness and readiness but low experience with online exams

The survey revealed that an overwhelming majority of students (95%) are aware of online assessments. Correlations were drawn between students who were unaware of online exams; and most originated from Tier III cities, and did not possess Internet connections at home. Even though awareness of online exams is extremely high among respondents, one of the most important considerations resides in the readiness towards trying the format, and perception towards universities who are currently holding exams online. ValueNotes found that the majority of students (69%) felt that these universities are more technically advanced, though students originating from Tier III cities are far more impressed with said universities. The majority of students (46%) consider the online process to be more secure, compared to traditional methods. There was also a general consensus that the accuracy and authenticity of online exams is far higher (68% of students agreed). There was a strong correlation drawn between these students, being more likely to join a university which conducts online exams. Awareness and readiness towards online exams aside, most students interviewed had minimal experience in giving exams online, at the university or entrance level. Further, the study found that the vast majority of

17

students (71%) who have given both entrance and university exams online are studying in Tier I cities, along with the majority of students (60%) who have given university exams online.

2.5.2

Perceived / expected benefits and issues

While most students are yet to experience the online exam format, they have strong opinions about the benefits they can expect from the system, over and above prevailing methods. These stated benefits were mostly consistent across both, university and entrance exams.

Exhibit 6: Perceived / expected benefits of online exams

100%

80%

60%

40%

20%

0% Convenience in applying Ease of taking exam Greater efficiency Higher confidence Transparency of Accurate and processes timely

Major benefit

Somewhat of a benefit

Not a benefit

Source: ValueNotes Research

Transparency of processes is seen as the major benefit, followed by greater efficiency. The convenience in online application/registration is perceived to be greater by 32% for competitive exams. However, students do not yet have a greater level of confidence with the online exam process, compared to traditional exams. The study also found that students are concerned with many aspects of the online exam processes, including security risks, the lack of infrastructure, the difficulty of use, followed by the lack of support from exam authorities.

18

Exhibit 7: Perceived / expected issues with online exams

100% 80% 60% 40% 20% 0% Security risks Infrastructure Difficult to use Support

Major issue

Somewhat of an issue

Not an issue

Source: ValueNotes Research

Comparing the perceived/expected issues for university vs. entrance exams, infrastructure is seen as more of a concern for competitive exams (62%), compared to university exams (34%). The majority of respondents (43%) who feel infrastructure is not a concern for online university exams are studying in Tier I cities. Further, the majority of students (39%) who claim infrastructure is a major concern for online university exams are studying in Tier III cities. There is little doubt that Indian higher education students are indeed aware of online exams, and most are ready to try the system. Though they do have some apprehensions about online tests, the predominant perception is definitely positive. This can be expected to increase in coming years, with greater Internet penetration, and further use of ICT in higher education. Hence, we believe that the university officials' perceptions about students' resistance to online exams (revealed in Chapter 4) are overblown. These may in fact be more likely used as a justification by those resisting the inevitable change in current systems.

2.6

Conclusion

Our research and survey findings present strong evidence that there is a need for online assessment exams in the Indian higher education sector. The respondents reacted positively to the idea, and almost 50% said they would want to start online exams in the next two years. Those who have already started would like to extend usage. This is because benefits of greater efficiency and costs savings of nearly 2050% are difficult to ignore. Students, the primary stakeholders are definitely aware of and ready to appear for entrance and university exams online. Their lack of experience in no way seems to adversely affect their opinion. Considering there is still a lot of groundwork to be done, the penetration of online exams will increase by 5 -10% in the next two years. However, given the greater levels of experiments and innovation in assessment we believe that there will be significant traction over the next three to five years.

19

3.
3.1

Importance of Assessment in Exam Management

Introduction

Traditionally, university functions range from intellectual (teaching and research), educational (cultivating minds, transmitting basic ideas) to social (bridging the two for social development), according to UNESCO. In order to effectively carry out these functions, the backbone of a university is its administrative activities. University administration is extremely challenging, with different colleges, departments and communities to coordinate; activities such as planning and policy making, curriculum setting, uniform course delivery and exam coordination. Among these, exam management is a vital area where universities need to extensively plan, implement and govern the many exam-related functions. The examinations process includes activities such as the registration of candidates, examination fee management, hall ticket generation, question bank management, question paper generation, evaluation, security control, processing and publication of results, re-evaluations and retests. Generally, it is the Controller of Examinations / Registrar-Exams (under the guidance of the Vice-Chancellor), who oversees the examination function at the university level. This includes:

conducting of examinations tracking examination expenditures and financial norms tracking academic calendars exam scheduling coordination and communication between different departments, affiliated colleges / students

Universities also conduct entrance exams for entry into their courses. Various exam boards (at the state/national/international level) oversee centralized admissions to colleges and universities / groups of institutes.

3.2

Typical exam process

Traditional exam management is a circuitous process and there is little that exam authorities can do away with. In addition, there are numerous administrative tasks at each level. Charted out below is a typical examination cycle. It is completely to the credit of the exam departments that expected outcomes are delivered, due to careful and adequate exam management. Each university and examination board has developed a system that fits their requirements. We have tried to capture the entrance and university examinations processes through the following diagram, and we believe similarities outnumber the differences.

20

Exhibit 8: Typical exam process

Generating examination forms Generating hall tickets Setting question papers Selecting final question papers

Collecting answer sheets back at distribution centers

Ensuring all question papers are delivered on time

Sending question papers to distribution centres

Printing

Storing them securely

Sending answer sheets for assessment

Collating all marks

Declare results

Get internal marks from colleges

Add to external marks

Revaluation and results

This part of the process is unique to university examinations

Source: ValueNotes Research

Most university officials agree that length of the process and time lines are concerns, yet nothing will allow for ineffective shortcuts or random experiments, as repercussions are huge. As is evident in Exhibit 8, the actual assessment activity takes up only about 1/5th of the process. Experts opine that any kind of interaction with a student can actually be treated as part of the assessment process. Assessment is a critical part of the exam process, as discussed in the next section.

3.3

Importance of measuring outcomes

Assessment is a core part of exam management systems, and is gaining greater importance in the competitive education segment. 'Assessment' refers to the systematic collection, review and use of information about educational programs undertaken for the purpose of improving student learning and development. (Marchese, T.J., AAHE Bulletin, 1987, 40, 3-8). Generally, assessment includes an examination of overall university goals and objectives, program evaluation, departmental effectiveness, and the evaluation of student learning. For the purpose of this study, we refer to the assessment of student learning as university 'assessment', including the planning, conducting, grading and evaluating of student examinations. Assessment of student learning (also known as outcomes assessment) demonstrates that the institutions students have knowledge, skills, and competencies consistent with institutional goals and that students at graduation have achieved appropriate higher education goals (Middle States Association, Characteristics of Excellence, 2002).

21

Exhibit 9: Interrelatedness of assessments to course curriculum objectives

Source: Assessment Resource Centre - modified from Biggs, J (2003). Teaching for Quality Learning at University, 2nd Edition, Society for Research into Higher Education & Open University Press

Assessment helps universities understand to what degree the 'Intended Learning Outcomes' defined for each course are met. The chart above illustrates the interrelatedness of assessments to course curriculum objectives. Assessment functions are vital for an academic institution, not only in adjudging the current quality of curricula and services offered, but also in planning long term quality improvement activities.

3.4

Types of exams and assessment tools

Educational institutes use a variety of assessment tools based on the type and purpose of each testing activity. The tools available to instructors enable them to rate students based on the following types of exams: Entrance exams These are given by candidates in pursuit of admission to

undergraduate/postgraduate/PhD courses at universities. These exams are either conducted by the university itself, or by an exam board representing a group of universities (which accept the scores), or by a national/state exam board. In-course exams - These form a means of gauging a student's understanding of the subject, and need for further guidance, improvement and feedback. Tools which aid in-course/mid-semester exams include classroom quizzes, tests with short answers, case studies and presentations. End-course exams - These are final verdicts of student performance, to determine the knowledge, skills and abilities imbibed by the student from the course. Students are tested on course material for the last time at the end of their semester(s)/year. The kinds of assessment tools which universities worldwide generally use are represented in the following table.

22

Exhibit 10: Types of assessment tools

Assessment tools Short answers Exam type In-course Pros Objective knowledge can be tested, encouraging recall (no prompts), easy to grade Easily scalable, easy to grade and compare scores Common queries can be addressed Analytical and synthetic thinking is displayed, with attention to presentation Team work towards curricular goals is encouraged Historic records of academic work reflect growth over time, encouraging self assessment Facilitates peer evaluation, understanding of course content is evident through presentation Knowledge outside of course content is brought forward, reflects analytical problem solving abilities Facilitates group learning and identification of learning gaps Depth of knowledge and critical thinking ability is tested Cons Prone to cheating and rote memory responses, limited testing of thinking and analytical skills Answers are predefined and limited in scope, prone to guesswork and cheating. No subjective answers Individual student attention is lost, prone to rote memory responses Highly subjective and difficult to grade

Multiple choice questions (MCQs) Quizzes Essays

Entrance / In-course In-course Entrance / End-course In-course / End-course In-course / End-course In-course

Team Projects Journals / Portfolios Oral presentations Case studies

Complicated grading due to varied team participation Difficult to grade due to size, varied subjects related to course content Presentations are challenging for some students, and grades may be biased towards better delivery rather than better content. Significant time required to build case study. Also prone to instances of plagiarism.

In-course

Peer reviews Debates

In-course Entrance / In-course

Peer evaluations made must be objective and based on evidence Time consuming for large classes. On the spot arguments may be stressful for students. Multiple evaluators reduce grading subjectivity. Level and extent of knowledge is untested

Checklists

In-course

Easy to grade according to established rubrics for passing each criterion

Source: ValueNotes Research

Most universities and exam boards worldwide make use of one or more of the mentioned assessment tools, based on the type of exam being carried out. The percentage of question answer format has reduced over the decades, to include other formats such as multiple choice questions (MCQ), match the following, short answers and one word answers. Some critics are concerned that students are limiting their learning and that the focus is shifting away from comprehensive learning. Yet innovations have continued and other assessment techniques such as projects, field work, classroom assessment, peer assessment, portfolio, reflective journaling, self-assessment, and performance-based assessment have been integrated into the system. Over time it has been proved that increase in assessment techniques actually enhances the learning process.

3.5

Trends and innovations in university assessment

Today, universities have become competitive enterprises. Competition in the global higher education sector has triggered the need for redefining much of the prevailing systems. Universities worldwide are battling for a share of the global student demand, experienced faculty, global rankings and funding. There is tremendous pressure from various stakeholders, including policy makers, accrediting bodies, and the

23

general public, to compete within this competitive environment. Universities must focus on their core functions in order to build competitive advantage. Innovative efforts must be focused on teaching and assessment, the core part of a university's functions. It must be noted that global trends in higher education also have implications for university functions such as exam management. These further impact assessment of student learning. Accordingly, the megatrends in global higher education, according to UNESCO (2009) and their impact on exam management and assessment are discussed below.

Exhibit 11: Megatrends in higher education and their impact on assessment

Megatrends in global higher education Globalization Internationalization Impact on assessment Reason Growing cross border education models Growing participation of employers Mass assessments through technology Innovations in assessment tools New teaching and assessment techniques are emerging to accommodate off campus learning. Employers keen on closing the skills-gap for fresh recruits are getting more involved in student assessment at the university level. With more students giving entrance exams every year, institutions have been forced to re-look at assessment methods to facilitate mass testing. Competitive universities are trying to differentiate the learning experience by innovations in assessment techniques such as specialized knowledge measures, rubrics, portfolios, external judgments of performance, performance evaluation other than grades, student interviews, and employer surveys. While ICT has also been instrumental in aspects of curriculum delivery and other administrative functions, its role is vital in driving improvements in student assessment.

The massification of higher education Student experience has become more complex, with more options, opportunities and challenges for universities to address The transformative role of Information and Communication Technologies (ICT)

The growth of ICT in assessment

Source: ValueNotes Research

Keeping in mind the scope of this report, we will focus on the role of technology in assessment, a concept that has more popularly come to be known as 'online assessment'. The Joint Information Systems Committee (JISC) of the United Kingdom has defined online assessment as, the end-to-end electronic assessment processes where ICT is used for the presentation of assessment activity, and the recording of responses. This includes the end-to-end assessment process from the perspective of learners, tutors, learning establishments, awarding bodies and regulators, and the general public. The next chapter will seek to explore and establish the global trends in online assessment, which are already redefining much of the higher education exam segment.

24

4.

Global Trends in Online Assessment

Today students seek to maximize learning from courses by extensively using technology and adding to the information already available through the Internet. This has also emerged as one of the top five trends in higher education. Awareness levels are up and there is greater willingness to experiment. Teachers and students share topical knowledge procured from the Internet. In fact globally, real time sharing and learning has gained momentum, with even social networking becoming a part of the learning process. According to a recent Australian study quoted on the UNESCO website, Facebook was the fourth most visited site. On an average, users spend 26.5 hours each week online, and a quarter of that time 6.5 hours is dedicated to Facebook. According to estimates, visits to another social networking site, Twitter, increased 1000% compared to the year before. Online assessment has become easier with greater computer and Internet penetration. Technology has improved further with faster and more accurate computers and applications. The global Internet penetration rate was pegged at 25.6% in 2009, according to Internet World Stats. With students being one of the largest segments with Internet access, universities across the world have a good reason to move to online assessment.

4.1

Online assessment: Increasing scope

As learning takes on myriad forms, so does assessment. In Chapter 3 we have covered various assessment techniques (please refer to Exhibit 10) that are widely used. The increase and variation in assessment techniques over the years was driven by the need to rate and assess students on various aspects of learning other than memory. These include the application of knowledge, thought flow, ability to process information, innovation and other softer aspects such as team work, creativity and ability to handle stress. Assessment is also treated as an input to redefine and alter teaching methods. Students that are found lagging are identified and provided extra attention. One key innovation has emerged in the form of online assessment. As defined earlier online testing is delivered with the use of computers on a network. The following exhibit explains how the Macquarie University in Australia implements various assessment techniques and their complete integration into the system. Instructions for the faculty indicate that there is scope for them to use online assessment as and when they require, with freedom to modify certain aspects. Similarly, it also allows them to retain control of essay type questions.

25

Exhibit 12: Online assessment process

Creating an assessment Create questions Create assessment Apply the assessment technique Question types True/false Multiple choice Combination multiple choice Fill in the blanks Jumbled sentence Matching Paragraph Short answers Calculated Assessment properties Title Question title Question delivery Duration Attempts available Submission settings Student score Show immediate feedback Result settings

Choosing assessment type Quizzes Self-tests Surveys

Generating assessment reports Performance report Item statistics Summary statistics Class statistics

Grading assessment Automatic grading for quizzes etc. Manual grading for essay type questions By student By Question

Assessment manager Grade assessment by students Grade assessment by question Re-grade questions Modify quiz tests View survey submissions Run reports

Source: Figure constructed with data from Macquarie University website

4.2

Online assessment: Increasing acceptance

As assessment becomes more student centric with an array of assessment formats, it is the faculty in institutes that will work to innovate and test different assessment methodologies. They also fill the critical role of subject matter experts (SMEs) and help to decide on the parameters needed for assessment. The following table explains the various formats in online exams in greater detail. From left to right (Columns 1-7), the options display a decreasing degree of constraint. For instance in a multiple choice question, students' responses will have to be one from those that are provided, while in a presentation, students will have the freedom to enter any responses. This table gives the likely array of options available in online exams along with an overview of how they can gradually move towards adoption, explains levels of complexity and gives options to innovate. Similarly, rows 1-4 depict the complexity of the tests. For instance, within the multiple choice questions, there are simple options such as true/false, and there can be slightly more complex options that enable alternate or multiple choice with media distractors.

26

Exhibit 13: Online assessment Questions and tasks

M os t Constrained Fully Selected Immediate Constraint Items Types Leas t Cons traine d Fully Constructed

Le ss Com ple x

1. Multiple choice
1A. True / False 1B. Alternate choice

2. 3. 4. 5. 6. 7. Selection / Reordering / Substitution Completion Construction Presentation Identification Rearrangement / Correction

2A. Multiple true / false 2B. Yes / No with explanation 3A. Matching 4A. Interlinear 5A. Single numerical constructed 5B. Short answer and sentence completion 5C. Closeprocedure 6A. Open-ended multiple choice 6B. Figural constructed response 6C. Concept map 7A. Project

3B. Categorizing

4B. Sore finger

7B. Demonstration, experiment, performance 7C. Discussion, interview

1C. Conventio nal or standard multiple choice 1D. Multiple choice with M ore new media Com plex distractors

2C. Multiple answer

3C. Ranking and sequencing

4C. Limited figural drawing

2D. Complex Multiple Choice

3D. Assembling proof

4D. Bug / fault correction

5D. Matrix completion

6D. Essay and Automated editing

7D. Diagnosis, teaching

Source: Figure reproduced from: Scalise, K. & Gifford, B. (2006). Computer-Based Assessment in E-Learning: A Framework for constructing Immediate Constraint Questions and Tasks for Technology Platforms. Journal of Technology, Learning and Assessment, 4(6).

Almost all online assessments are a combination of at least some of the options presented above. Over the years online testing became more pan-optic in its offerings it has grown from MCQs to include drag and drop, rearrange statements or pictures, fill in multiple gaps, diagrams, essays and others. It is vital that the learning and assessment strategy is shared with students and faculty to increase overall efficiency. Understanding the delivery mechanism and expected outcomes will make online assessment more effective and easier to adopt. Several studies and surveys have revealed the readiness of students to appear for online tests. Surveys mapping experiences in controlled environments revealed issues pertaining to difficulty in usage, but these are manageable challenges.

4.3

Use of technology in assessment set to grow further

Tech-savvy teachers are increasingly using blogs, edublogs to be more precise, for real-time assessment. As said earlier, it is important for the faculty to adopt and encourage the use of technology, as it forms an important part of the delivery mechanism. Web 2.0 is making its presence felt and will impact classroom assessment most through real-time, collaborative learning opportunities.

Expensive authoring tools are proving to be a concern, especially for exams that need to go beyond objective type questions. While this is a big challenge for exam authorities, constant innovations

27

and improvements in authoring technologies that have been witnessed over the last few years will cater to these requirements too. The growth of open source learning applications offer a cost effective means for universities to experiment with online assessment.

Penetration of mobile phones has increased, with a rise in smartphone adoption. This results in greater accessibility to learning and assessment tools, irrespective of geographical differences

4.4

Diverse applications of online assessment

Online assessment is currently used more for formative tests, which include online quizzes and discussion forums. Universities are encouraging faculty to store question banks online as almost all online assessment solutions offer random selection of questions to administer to students. This reduces the time that faculty needs to spend on preparing examination papers, especially for classroom and internal assessments.

Summative online assessments (end-course exams) are taking a little longer to move online, due to issues such as security, authentication and access. Technology/learning companies are offering solutions with promises of completely secure environments, and more and more universities are experimenting with these. Proctoring is however still highly prevalent, as it allows for greater security, especially for students not appearing for tests on campus. A number of universities have moved to online submission of written assignments. This reduces the administrative tasks immensely. With almost 50% of the assessment cycle consisting of administrative tasks, the
- Keith Hampson, Digital Education Strategies, Ryerson University, Canada Technically, every kind of assessment can transition to an online format. The implications are primarily financial. As an example, engineering and mathematics students who need to demonstrate in practicals will need authoring solutions, which are very expensive. I think exam formats vary with differences in disciples and types, perhaps 5-7 years down the line, we will be able to see the full applicability of online assessments in these fields.

adoption of assessment technologies can bring in large gains for administration.

Distance and open education are likely to completely move to the online format the fastest. The structure of these universities is most suited for adoption of online assessment. There is great synergy between the two, with emphasis on anytime, anywhere education.

4.5

Customized as per regional needs

Universities in the developing world are still struggling with access to knowledge, infrastructure and acceptance. The growth is slow. This is of course not to say that efforts are lacking. The volumes are so large that it definitely makes complete economic sense for universities to go online and this is reflected in the number of universities that are running pilots.

The primary driver to adopt online assessment may vary depending upon the requirements of a region. For instance rural areas where concentration of population is low, have found that students prefer online assessments so that they do not have to travel.

28

4.6

Examples of online exams

4.6.1

Utah Education Network

The Utah Education Network (UEN) uses a course management system that prevents students from printing, copying, visiting other URLs, or accessing other applications during an online assessment. Students are essentially locked into the assessment until they submit it for grading. This software is installed within testing labs or on student computers.

4.6.2

Graduate Record Examinations (GRE)

Graduate programs and business schools use GRE scores to evaluate readiness for graduate-level studies. GRE measures verbal reasoning, quantitative reasoning, critical thinking and analytical writing skills. Every year more than 600,000 prospective graduate school applicants from approximately 230 countries take the test in CBT (computer based test) centres. The paper-pen test format is still used in countries that cannot offer CBT facility. Online registration is available for all students.

4.6.3

Test of English as a Foreign Language (TOEFL)

The 'Test of English as a Foreign Language' (TOEFL) is a globally recognized exam which enables candidates to validate their English language skills (reading, listening, speaking and writing), when applying to universities. More institutions accept TOEFL test scores than any other test scores in the world more than 7,000 colleges, universities and licensing agencies in more than 130 countries. The TOEFL test can be taken online, though the paper-pen format is available in countries where the online version is not supported. Candidates can register online and get an immediate confirmation of time and location of the exam. The website also offers online learning and practice tools. Further, the CBT test allows candidates to take notes, review answers, highlight unanswered questions, and manage their available time. Once the exam is complete, students get faster score reporting online, with diagnostic feedback for specific skill development. The testing agency, ETS, claims the Internet based test is more cost effective and offers a better testing experience.

4.6.4

Graduate Management Admission Test (GMAT)

The Graduate Management Admission Test (GMAT) is a computer adaptive examination to test verbal, mathematical, and analytical writing skills. The GMAT uses individual computer workstations to administer the exams. Computer adaptive tests means that at the start of each multiple-choice section, examinees are presented with a question of medium difficulty. As they answer each question, the computer records the scores. The computer then uses the responses and the test design to decide the sequence of further questions. With correct responses, questions of increased difficulty appear. In case of incorrect answers, questions of less difficulty level appear. GMAT scores are used by applicants for advanced study in business and management. It is administered around the year at test centres across the world. The scores are used by approximately 4,700 graduate management programs at more than 1,900 institutes.

29

5.
5.1

Indian Higher Education: An Overview

Indian higher education: A snapshot

Indian higher education has become a topic of mainstream discussion over the last year. The reformist minister, Mr. Kapil Sibal has initiated a lot of discussion with his attempts to bring about changes in the Indian education system. With the media and general public adding more pressure for reforms, the education system looks set for a complete overhaul. The stakeholders, namely universities, exam boards, educators and students must utilize this opportunity to bring about the long sought changes. In this chapter we will look at the prevailing scenario and analyse the various parameters that impact the sector. The Indian higher education system is the third largest in the world. The following diagram represents the structure of the Indian higher education system.

Exhibit 14: Structure and size Indian higher education

Central Universities

Central Government

Institutes of National Importance Deemed and Private Universities

AICTE, MCI, ICAR, NCTE, DCI, etc.

Indian higher education: Quick facts

Number of Universities in India: 360 Number of Institutes of National Importance: 33

Central Advisory Board of Education (CABE)

University Grants Commission

State Universities Private Universities

Number of colleges in India: 18,064 Number of students enrolled: 14 million

State Government

Source: Department of Higher Education website, Government of India

Source: Department of Higher Education, Government of India

The Central Government is responsible for major policy relating to higher education in India. It establishes central universities, provides grants to University Grants Commission (UGC) and is responsible for declaration of Deemed to be University on recommendation of the UGC. The State Governments are responsible for establishment of state universities, and provide grants for their development and maintenance. The coordination and cooperation between the Central and State Governments in the field of education is through the Central Advisory Board of Education (CABE). Education is on the Concurrent list subject to Entry 66 in the Union List of the Constitution giving exclusive Legislative Power to the Central

30

Government to co-ordinate and determine standards. UGC is responsible for coordination, determination and maintenance of standards and release of grants. National Assessment and Accreditation Council (NAAC) is an autonomous institution established by the UGC in 1994 to assess and accredit institutions of higher education that volunteer for the process, based on prescribed criteria. Professional councils responsible for recognition of courses, promotion of professional institutions and providing grants to undergraduate programmes include All India Council of Technical Education (AICTE), Medical Council of India (MCI), Indian Council for Agricultural Research (ICAR), National Council for Teacher Education (NCTE), Bar Council of India (BCI), Central Council for Indian Medicine (CCIM), Distance Education Council, State Councils of Higher Education, etc.

5.1.1

Budget allocation and plans Big but inadequate

The Indian Prime Minister Dr. Manmohan Singh termed the XI Five Year Plan as 'India's educational plan'. The XI Plan made education the highest priority to achieve fast and inclusive growth.

Exhibit 15: Budget allocation for higher education Year

2005-06 2006-07 2007-08 2008-09 2009-10

Budget Allocation (in Rs. billion)

58 70 92 113 154 -

% increase

21 31 23 36

Source: Department of Indian Higher Education, Government of India

As per the Annual Report 2008-09 of The Department of Higher Education, the XI Plan allocation was Rs. 849.43 billion for the Department of Higher Education.

According to the union budget 2009-10 proposals, the higher education segment will receive Rs. 154.29 billion including a planned allocation of Rs. 96 billion. This is an increase of almost 36% over the last financial year, when the total allocation was Rs. 113 billion.

The Government also allocated Rs. 8.27 billion to achieve its goal of establishing one central university in each uncovered state.

Other initiatives include opening of 30 new universities, eight new Indian Institutes of Technology (IIT), seven Indian Institutes of Management (IIM) and 20 Indian Institutes of Information Technology, five Indian Institutes of Science Education and Research, two schools of planning and architecture and ten National Institutes of Technology and 1,000 new polytechnics.

For each of the universities and colleges that are to be set up, there is a need for teachers, equipment and infrastructure, which is a challenge even for existing ones. Implementation at the grassroot level is very poor and unless that improves, it is unlikely that the situation will improve. The report on the 55th meeting of the CABE recommended, Since government financing will remain the cornerstone, government support for higher education should increase to at least 1.5 per cent of GDP, out

31

of a total of at least 6 per cent of GDP for education. Despite the large budgets, given the huge population, this will not be enough. In recent years, we have seen the advent of a fast growing, professionally managed private education sector which is exploring opportunities due to inadequate spending and poor quality of government institutes.

5.1.2

A huge demand-supply gap

According to the XI Plan document, by the end of the Tenth Plan, the Indian higher education system had 378 universities, 18064 colleges, a faculty strength of 492,000, and an estimated enrollment of 14 million students. The higher education institutions include 23 Central universities, 216 state universities, 110 deemed universities, 11 private universities, and 33 institutions of national importance established through central legislation and another 5 institutions established through state legislation.

With only 11% of the eligible age cohort enrolling in higher education, India is way behind the global average, even when compared to other developing countries. According to estimates, India needs nearly 1,500 universities to achieve its Gross Enrollment Ratio (GER) target of 15%. The graph below depicts the difference between India and other countries and regions such as the US and Canada with 60% GER and the other BRIC countries at 21%.

Exhibit 16: Comparative Gross Enrollment Ratio

Gross Enrollment Ratio (in %) 60

55

36.5 23 11 22

India

World average

China

Other developing countries

Developed countries

US

Source: ValueNotes Research

The other major gap is in the number of teachers. With nearly 500,000 teachers, it is very important to ensure that the number and quality of teachers improves. According to estimates, India needs 1 million teachers to cater to the current demand.

5.1.3

Higher education: Quality issues

Lack of teacher training has been established as one of the major causes for higher education not taking off in India. With the rate of PhDs in India at less that 1% every year, it is extremely difficult to employ teachers. Numerous posts are kept vacant for long periods and those teachers that are employed may not necessarily be suitably qualified to teach students.

Many universities work with inadequate infrastructure and students do not receive the facilities that

32

they are promised. The recent imbroglio over Deemed Universities is a case in point. Similarly, the State of Chhattisgarh witnessed the mushrooming of private universities a few years ago. Students took admissions without checking for information on the status of these universities. When many of these were later de-recognized, students were left in a lurch.

The lack of quality is also apparent as many students cannot find employment on graduation. Indian and multinational employers complain about the 'unemployability' of the Indian student population. Some surveys suggest that as little as 25% of technical graduates and 10-15% of other graduates are, in fact, employable on course completion.

To tide over these issues, NAAC has been established as an accreditation authority, that will closely follow the Universities' performance and ensure compliance.

5.2

Going forward: Indian education needs quantity and quality

The education segment is greatly affected by the political, economical, socio-cultural and technological environment that prevails in the country. Indian higher education needs to make efforts to meet not just the demand but also the quality. This section will discuss various issues that are likely to impact the segment. It is important that an all round effort be made for the higher education segment to thrive and meet the expectations.

5.2.1

Driving changes at the policy level

5.2.1.1 New bills and regulations

The current Central Government is planning to introduce three bills for higher education. These bills can be expected to raise current quality standards in existing universities, and curb malpractices.

Plans are afoot to establish a National Commission for Higher Education and Research (NCHER) as an overarching body in higher education. This body will propose accreditation norms and standards. The decision is based on the recommendations of Yashpal Committee and the National Knowledge Commission (NKC).

The Ministry proposes to set up The Educational Tribunal Bill to provide for a three-Tier system district, state and national level - to settle studentinstitution and teacherinstitution disputes.

The Prohibition of Unfair Practices in Technical, Medical Educational Institutions and Universities Bill will empower the Centre to curb unfair practices enabling them to impose penalties and fines against deviant institutes. The issue of 'fake universities' will also be curtailed with the help of this bill.

5.2.1.2 Increasing competition

The government is planning to allow foreign universities to enter the Indian market. Since India is an extremely lucrative market, a number of globally renowned universities have already expressed interest in establishing branches in India.

33

Similarly, greater freedom to private players, coupled with sufficient regulation, will also help improve the quality of higher education in India. Public-private partnership is an option that is being explored by a number of corporate houses.

The Planning Commission has identified a resource gap of Rs. 2.2 trillion, which is difficult for the Government to address on its own. This makes it imperative that the private players play a greater role in the segment. According to Venture Intelligence, the Indian private education market in 2008 was pegged at $3 billion.

The entry and proliferation of private players has already heightened competition for the sector, as they peg themselves against global competition. The quality of education in line with international standards is given greater emphasis, as even students are now looking for global employment opportunities. As foreign universities set up base in the country, competition in the sector will positively impact the quality of higher education nationally.

Higher Education in India is gearing up for qualitative leap frogging. While some universities have the mind set of rapid change others wish to maintain status quo. Latter are a great liability for the Indian higher education. India is gradually becoming a destination for higher professional education for not only the third world countries but Australia and Europe is also going to send lot of learners. - Prof.B.K. Kuthiala, Vice Chancellor, Makhanlal Chaturvedi Rashtriya Patrakarita Mahavidyalaya

5.2.1.3 Need for consistency and deregulation

Government policies keep changing, as each new political party that comes to power puts in place a new set of rules. This does not allow the education segment to have long-term policies.

Decision making is not deregulated. Even functional and operational decisions are taken by the government resulting in many delays, especially in central and state universities. Power at the grassroot level is not sufficient, and therefore student welfare is inadequately addressed.

For higher education to go up by another notch, it is important that all stakeholders get involved in the decision making process. Lack of authority impacts quality and constant policy changes also dilute the benefits of earlier policy changes.

5.2.2

Demand will drive growth

5.2.2.1 Rising middle class driving demand

The demand for higher education institutes will increase, as the number of students is expected to increase at the rate of 5% annually. The National Knowledge Commission has recommended 1,500 universities in the long run.

Public expenditure on education has increased, however the percentage share of Gross Domestic Product (GDP) spent on higher education has reduced from 0.77% in 1991 to 0.7% in 2008, as per an E&Y-FICCI report.

Banks in India have identified education loans as a big potential market valued at more than Rs. 300 billion and almost all banks have it as part of their portfolio.

The conversion rate of students opting for higher education after high school is dismal. Since there seems

34

to be a consensus on increasing government allocation to encourage students, schemes such as scholarships need to be encouraged.

5.2.2.2 Interest in the sector rising

Corporates are encouraging employees to learn as they earn, thus increasing the life-long learning market.

In 2009, the education sector received $118 million from venture capital and private equity investments across nine deals, and this trend is expected to continue in 2010. The education market is estimated to be worth $80-billion, and the number of private equity deals has increased four-fold in the last four years.

The Government needs to act fast in exploring the options of maximizing public private partnership in higher education and implementing changes in policy. While there is still opposition to make it a profit generating endeavour, if regulations and strict monitoring are adhered to, the entry of private players will provide this sector with the much needed impetus.

5.2.3

Gender parity and greater awareness drives growth

Indians are giving greater importance to higher education, as an increasing number of employment opportunities are on offer. Parents across social strata now see education as the best means to enhance socio-economic status. This is particularly relevant for the growing Indian middle class.

Corporate culture and globalization have changed the mindsets of people. Ongoing / continuing education is gaining importance, as people are open to learning alongside employment. Corporates and education institutes are tying up to offer learning opportunities.

A higher number of girls in the country are now looking at the option of higher education. Central and State Governments are also encouraging female education by waiving tuition and exam fees at junior college, undergraduate as well as postgraduate levels.

5.3

Indian higher education: long way to go

5.3.1

Planning and policy makeover

Indian higher education needs uniform policies, and yet localization is the key. While it is important that education across the country adheres to uniform regulations that will help in maintaining quality, it is equally important to localize as per the requirement of each state. For instance, some states are way ahead of the curve in terms of inclusion of ICT, while others have a lot of catching up to do. This is needed so that each university can decide on its requirements and work accordingly. While private and deemed universities are free to adopt changes as and when they see the opportunity, many central and state universities have circuitous decision making mechanisms. Policy disruptions due to government change can be avoided with a central committee with industry representation, education experts and

35

university officials which can streamline and follow up on the plan.

5.3.2

Higher education needs the ICT push

The UGC needs to encourage universities to increase the rate of adoption of ICT. The incorporation of ICT is imperative for the sector to become globally competitive. For e-learning to make a mark, it will have to be monitored closely. Moving courses online, will not suffice and there needs to be a constant follow up and sharing on best practices, learnings and quality so that students get the maximum value out of it. To retain the quality of education, the education segment players need to commit themselves to certain ethics, which offer equal opportunities to all and ensure that admissions are only on merit. ICT can help in this initiative.

5.3.3

Survival of the best

performance is assessed at every step, yet

Students'

universities, colleges and teachers function without any kind of assessment at all. Strict regulations on quality are imperative. Entry of private and foreign universities will change the way the entire system functions as they will be adequately equipped to force bad quality institutions to improve, or else, go out of business.

Our students are observing online exam facilities in deemed and private universities. They are even dropping out half-way to join those institutions, because we have not yet incorporated technology in learning and assessment. - COE at a state university in Gujarat

5.4

The Indian exam market

Given the diversity and complexity of the Indian education system, reliable and comprehensive numbers on the size of the exam market of the number of exams conducted are simply not available. Based on our discussions with experts and decision makers at the universities, along with published data from the Government and university sources, we have tried to derive a reasonable estimate of size and value. For ease of analysis, we have separated the Indian higher education exam market into two parts.

University exams which include mid term, final semester/annual exams, Entrance exams, held by exam boards and universities at city/state/national level

We used published data supported by expert opinion to arrive at reasonable assumptions for the following:

Number of students who appear for exams Number of universities that hold entrance exams Number of entrance exams held by universities Number of various state level exams Average costs for various kinds of exams

Based on our calculations, the market size for the Indian higher education exam market, by type of exam,

36

is as follows. Here we have considered only the end-course exams and entrance exams at various levels. If we were to include the in-course exams the figures would increase by approximately 12%.

Exhibit 17: Market size of Indian higher education exam market Type of exam
University semester exam National level entrance exam State level entrance exam University entrance exam

Market size low range (in Rs. million)

4,100 9,600 1,100 20,500

Market size high range (in Rs. million)

6,500 13,400 1,500 31,000

Total size

35,300
Source: ValueNotes Research

52,400

The Indian higher education segment has nearly 14 million students. Taking into consideration the average number of exams and the cost of exams, we estimate the university exam market to be in the range of Rs. 4,100 6,500 million. We believe this to be a conservative estimate, since we have not considered:

in-course assessment re-evaluations re-tests any form of informal assessment different fee structures and costs for different categories of students exam facilities in foreign locations

Entrance exams on the likes of IIT-JEE, GATE, MHCET are held for entrance to professional courses by respective universities and national/state exam boards. Again, based on estimates of number of exams and students who appear for these entrance exams, we believe the value of this market is in the range of Rs. 31,200- 45,900 million. No doubt a conservative estimate, as even for the higher figure, we have not considered certification exams such as CA, CS, and others. We have also calculated the number of exams in the country to be over 200 million each year, including end-course and entrance exams. Needless to say, the potential is huge. Given the above, the actual market opportunity is likely to be close to the high-end of our estimate range. Further as we have seen, there is considerable growth in higher education and enrollments are set to rise, at a CAGR of 8.5%, according to National Institute of Science Communication and Information Resources. Further, more and more applicants appear for Indian higher education entrance exams every year. As an example, the CAT exam (taken by 250,000 students in 2009) witnesses an increase of nearly 40,000 candidates every year. Accordingly, it can be expected that market will grow by about 10% per year, as costs are rising faster. These figures certainly justify the need for a step up in efficiency in many areas of exam management. Automation and the use of technology in exam management could save universities and exam boards

37

around 20-50% of their overall costs, depending on the level of implementation. This is an essential task in the cost-benefit analysis, which will influence universities to go online. However, with rising expenditures on one side, and cost and efficiency gains from technology on the other, it is safe to say that technology will play a major role in the Indian higher education exam market in coming years. Given the huge size of the market and the clear cost and efficiency benefits, we believe penetration will increase dramatically over the next five years.

5.5

Challenges in traditional exam management

Universities today face challenges in many aspects of exam management, for entrance as well as semester exams. These challenges typically can be classified as administrative, resource and design.

Exhibit 18: Challenges in university exam management

Challenges in university exam management

Administrative challenges Registration Security and impersonation Valuation and scoring Declaration of results Exam logistics

Exam design challenges Level of difficulty Importance of internal and informal testing Designing for mass testing

Resource allocation challenges Budgetary constraints Availability of qualified examiners

Source: ValueNotes Research

The following section discusses these challenges briefly.

5.5.1

Administrative challenges
Registration Universities with hundreds of affiliated institutes must distribute physical prospectuses and application forms to all locations, and this becomes increasingly difficult to manage for nation-wide entrance exams. Generating and distributing hall tickets on time is another challenge faced in the traditional process.

Exam logistics Smaller universities usually face problems of scale, especially entrance exams conducted at multiple locations. Exam venues are also an issue when exams are held in multiple

38

batches. There are often errors in assigning batches, and a lot of confusion arises around exam timings for individual students.

Security - Apart from this, the major concerns for universities worldwide is security, especially the many instances of cheating and misrepresentation among students. Other forms of cheating include involvement of examiners, paper leakages, etc. Universities spend much time and effort in combating this challenge, though it is one which appears across any medium in use. Two final year MA students of a reputed university who were waiting for their results were asked to reappear for an exam after a gap of two months, as their answer sheets were lost. As a result they were not able to appear for the M Phil or PhD D selection exams. After 500 students were found copying en masse at one center in Uttar Pradesh, the exam officials cancelled the exam in 2009.

Impersonation There are many instances of student impersonations in large universities, which rely on external examiners to supervise exam execution. Entrance exams are more prone to such malpractices. As an example, five candidates appearing for the Uttar Pradesh Combined Premedical Test (CPMT) 2009 were detained for impersonating registered candidates. Other recently reported cases of impersonation included Jammu and Kashmir Combined Entrance Test (CET), and the Postgraduate Entrance Test (PGET) conducted by the Rajiv Gandhi University of Health Sciences (RGUHS).

Valuation and scoring Universities generally appoint officials in charge of students' exam transcripts for each exam centre. This allows them a certain degree of control and accountability in case of any issues. The challenges are mostly in the form of time and effort required to physically manage transcripts and send them across to assessment centres. Government regulations across countries do call for centralized assessment programs, to smoothen the process. However, this may not hold true for private universities. The issue of uniformity crops up when it comes to assessment, as essay based, subjective exams cannot be evaluated in the same manner by different examiners. To counter this, universities often have to strengthen the process by introducing two levels of assessment per paper.
Co-ordinating papers for assessment at various central assessment centers, calling the examiners for assessment and getting the work done in stipulated time is a big issue. Co-ordinating all of this takes significant time. - Dr. Udayraj Raorane, COE, Maharashtra University of Health Sciences

Declaration of results Result declaration is an area that directly affects students' future careers, especially final year results. While state governed rules do exist, results are sometimes delayed, and are unpredictable in terms of release date. Students are most often not informed and authorities fail to specify accurate declaration details. The consolidation of marks is delayed due to inconsistencies in internal marks, delays in second round of assessment, disputes in merit list creation, etc. When university admissions depend on entrance exam result, there are many pressures that exam departments face from faculty and students alike. Students of a prominent university who appeared for exams in April-May 2009 had not received their results till October 2009, which put the provisional admissions of many students to other universities in a jeopardy.

39

5.5.2

Exam design challenges

Level of difficulty Students from the same course need to be tested at a uniform level of difficulty, so that comparative ratings are possible. Students are given different exam papers, and there are complaints about differing levels of difficulty in exams. To counter this, faculty draw up question banks with different levels of testing, and choose similar questions from each level to place in different question papers. However, the perception of some papers being 'much easier', 'unfair' and 'lucky' still remain in the minds of students.

Importance of internal and informal testing Some universities depend on semester/final exams for indications on student performance. The importance of internal and informal testing is undermined, as internal assessments are not always formally counted for final scores. Faculty are also unaware/uninterested in the different ways in which informal testing can be conducted for students, and expect final written exams to account for all learning.

Designing for mass testing Entrance exams aim to select students that meet certain standards with respect to knowledge, skills and abilities. However, evaluators often fail to test any skills and abilities, and knowledge testing through objective type questions fail to capture aspects of creative and writing abilities. This is a major challenge for university exams as well, but an inevitable one. The level of scale involved in entrance exams is tremendous, and a fast turnaround time for assessments can only be delivered with objective assessment tools.

5.5.3

Resource allocation challenges

Availability of qualified examiners There is a gross shortage of qualified lecturers, readers and professors. The percentage of unfilled teacher positions in state universities in India stands at 58% today, even higher for lecturers and readers. With rising student population, the supply-demand gap is also increasing. Internal assessments are delayed, final papers uncorrected, and the processing of marks stalled. Ultimately, the timely declaration of results suffers greatly. This is especially so for niche courses (for example, medical super specializations). Also, the examination schedules of most universities coincide, creating a resource crunch particularly around exam time.

A newspaper report cited the case of a university in western India, where students could purchase the names of examiners who were due to check their papers, along with their postal addresses for Rs. 500. As mentioned in the article, with so many people involved, it is difficult to ensure the integrity of each person. Also due to lack of qualified manpower examiners who do not measure up to the requirements are also used. Budgetary constraints Publicly funded universities face many challenges due to budgetary constraints. Entire exam fees are waived or heavily subsidized, and the university has limited grants to maintain current exam processes. Any investment or upgrading in technology to support administrative tasks is a challenge.

5.6

Tackling exam challenges through technology

The Indian market is unique given the diversity and the number of students, infrastructure issues, and other

40

challenges such as shortage of qualified teachers, and regional imbalance. These challenges also affect assessment directly and efforts to use technology to deal with some of them are already underway. Universities have only started using online assessment over the last 5-6 years. As per our survey, 13% of respondents are already using the online format. The pioneers in conducting entrance exam online, are Manipal University and the Birla Institute of Technology and Science (BITS), Pilani. They have been continuously evaluating students online for last 5 years without a hitch. Similarly, there are many other universities that have started experimenting with online assessment. Year 2009 was probably the most dramatic when it came to online exams. Following are some of the tasks that are being managed better with use of technology, and also some successful attempts that have been made.

5.6.1

Increase in experiments to increase assessment effectiveness

Experiments with assessment techniques The policy makers have realized the need to reform assessment system. The UGC has started issuing directives to try and initiate constructive changes. Efforts at individual university levels are also underway. Universities are experimenting by mixing assessment techniques and using projects, objective, field work, corporate experience and others to improve overall quality of assessment outcome by and make it more robust.

Use of both formats - A number of attempts at using both online and paper-pen examination formats are also underway. Though complete move to the online system is delayed, it allows those experimenting a surer footing and greater confidence going ahead.

Gujarat Technological University takes entrance exams online

Objectives: Conduct entrance exam online

To make the process faster To make the process more transparent

Benefits:

The entire process from application processing to online counseling for seat allocation was online.

Declaration of results in two hours added to credibility and transparency of the entire exercise.

More than 22,000 students appeared and each district was equipped with two 50seat test-centers.

Copying was curtailed with a different question paper for each student.

The director of GTU Manish Bharadwaj was quoted in Times of India as saying that instead of the usual start of new batches in August, the online exams enabled completion of the entire process much faster and would result in the academic session starting in July.

41

5.6.2

Managing with less resources

Decentralization of processes - Exam management authorities are looking at methods that will reduce the length of the exam process without diluting the impact. Departments and colleges are being allowed more control, especially in the mid-term tests and other assessments. This also eases the pressure on the process, and the scale is reduced. Though not followed widely, colleges taking over mid-term exams is gaining ground and at the end of the term, university exam authorities receive collated marks and can manage it easily.

Reduces pressure on administration - Many universities including those that are not comfortable with using online exams are instead initiating online registration and application for students. This has enabled the automation of repetitive and mundane tasks that do not need human intervention. It has also translated into lower manpower needs and lower costs during the process.

Case study: NMIMS move to online exam

Objective: As stated by the vice-chancellor, Rajan Saxena in DNA, the objective was:

to facilitate students appearing from the UK and other Asian countries to make handling of close to 50,000 applicants easier to make the process more uniform, transparent and reduce malpractices

Benefits: Narsee Monjee Institute of Management Studies (NMIMS) opted for online entrance exams in 2009.

Anupam Rastogi, admissions chairperson, NMIMS said the move would generate greater number of applications, indicating students prefer online format.

The window of 10 days allowed students to reschedule their exam dates to accommodate the 2nd round of CAT dates.

5.6.3

Rising importance to security and controlling malpractices

Using technology to improve security Some measures include bar-coding answer sheets to ensure complete security. Online exams ensure better security and multiple measures to curb malpractices such as student misrepresentation, cheating, etc.

Case study: Manipal University - One of the front-runners in online exams

Objective: The drivers to move to online exams were:

To establish itself as a forward looking university. To allow flexibility to students to select their entrance exam date and location

Benefits:

Ensures flexibility to students and they can appear in multiple exams.

42

Also takes care of the challenge of a fair examination that is devoid of all malpractices.

Manipal University allows candidates the flexibility to schedule tests at their own convenience. The online entrance tests are held over a pre-specified period of time. The test centers are located across India and have computers for all students. The electronic delivery of tests ensures high security and use of bio-metric and hi-tech monitoring system is used to prevent impersonation.

Efforts to increase transparency Many universities have allocated a small percentage of overall assessment to attendance, and now they are taking a step further with online attendance and biometric attendance, so that all stakeholders have access to real time information. Contrary to traditional methods, transparency in the system is now becoming an asset, increasing student satisfaction.

Case study: University of Pune looking for better practices

Objective: The drivers for attempting online exams were:

the need to conduct free and fair exams, and reduce the burden on the examination department with an efficient and effective system.

Benefits:

Nearly 20,000 students from 57 engineering colleges affiliated to the University of Pune appeared in an online exam in 2009.

This Compact Disc (CD) based examination made the system more transparent, simple, and reliable enabling faster declaration of results.

Prakash Wani, dean of the engineering faculty told Indian Express, "This will help in reducing time required for assessment work, thereby assisting the university authorities to declare results well within the time in spite of increase in number of students and colleges. There is no burden on the students to apply any more for photocopies and revaluation. After the success of this venture, UoP plans to further extend this to more subjects and 11 more colleges.

43

6.

Technology in Indian Higher Education Exams

ValueNotes conducted a survey of decision makers in the Indian higher education sector. In our survey we included Vice chancellors, Director, Deans, Registrars, and Controllers of Examination from universities and Institutes of National Importance from across India. Annexure 1 (pg. 79) lists the participating institutes and universities. We talked to about 75 respondents from 55 Universities/ institutions. Other details related to the respondents are as follows:
Respondent Breakup Vice Chancellors Controllers of examination Registrars Directors Experts Others University / Institute breakup State/Central/Open Universities Deemed/Private Universities National Institutes of Importance Exam boards 49% 38% 5% 7% 20% 31% 22% 7% 12% 8%

The respondents shared with us their experiences, expectations and opinions on various issues related to the higher education. The survey, the first of its kind was to assess the acceptability, readiness and experiences of respondents. In the following segments, the survey findings reveal the intent of respondents and the impact on execution followed by the perceived benefits and challenges.

6.1

Online examination: Intent

6.1.1

High interest in conducting exams online

Increase in interest with growing awareness The interest level of the respondents is extremely high at 52%. Together with those who have already experimented or are experimenting, it displays an extremely positive trend. Only 16% of the respondents are not interested in using technology for online assessment. The rest have expressed an interest and are trying out various options.

44

Exhibit 19: Interest levels in online exams are high

15% 4% 4% 7%

16%
Not interes ted and not tried Interes ted but not tried Still experim enting Tried/experim ented but decided not to go ahead Tried/experim ented and decided to go ahead Already us ing

52%

Source: ValueNotes Research

6.1.2

Future plans

Our survey indicates the positive attitude of the decision makers with respect to online exams. Opinions and reasons are numerous. The current focus is predominantly on process related issues. About 68% of the respondents said that they plan to either increase the current extent of online exams or introduce it. This overwhelming majority represents the willingness to try online exams. Almost 2/3rd of our respondents saying said they want to implement online exams in the future. Going forward, Indian higher education needs to be equipped with trained teachers, infrastructure and technology that can adapt to Indian requirements, which in turn calls for a lot of experimentation, innovation in assessment techniques, technology upgrades and readiness to adopt changes and failures.

Exhibit 20: Moving towards online exams?

9% 2%

Plan to increase / introduce online assessments Do not plan to increase / introduce online assessments Considering increasing / introducing online assessments Can't foresee
Source: ValueNotes Research

21%

68%

45

6.1.3

Lack of clarity

However there is lack of clarity on when the plans may actually be executed. With almost half the respondents believing that less than 10% exams will be online in the next 2 years, and almost 50% looking to go online in the next two years, the gap or opportunity is tremendous. It establishes the need to share knowledge, experience and beliefs about the entire process of going online. It also firmly establishes that improvement in assessment with the use of technology is the way to go. Technology providers and decision makers have to work together to help the proponents in the Indian higher education segment make this transition to online system in the next 6-36 month period, to convert the remaining 30% who have not set any time frame for themselves.

Exhibit 21: Lack of clarity on implementation time frame

25% 32%

14% 18% in 6 months in 12 months in 24 months

11% in 36 months Can't foresee

Source: ValueNotes Research

6.2

Online exams: perceived benefits

Most benefits of online exams that were revealed in respondent interviews are in line with trends in exam management. Therefore, the likelihood of greater acceptance and implementation in the next 2-4 years is evident. The perceived benefits represented in the graph below have a direct impact on execution. All of these, as discussed in the previous section, will help in better execution of exams which will impact the time, the processes and the outcome. Respondents' opinions on lowering costs and other benefits are discussed below.

46

Exhibit 22: Online examinations many potential rewards

Faster results Greater transparency Technology led student authentication Uniform assessment Higher security Manpow er savings Paper free, greener process Shorter exam cycle Overall low er cost Increased geographical reach Perception of progressive university (by students) Convenience for students (batches) 0% 20% 40% 60% 80% 100%

Very important

Somewhat important
Source: ValueNotes Research

Less important

6.2.1

Benefits: Lower costs

A lot of environmental problems would be addressed if we moved to paper free system, not to mention cost. There is so much stationery consumed during exams . During March-April, when 70% of exams are held, 3 lakh answer books are used, at Rs.40 each. This is nearly Rs.1.2 crores. And these are just the main answer sheets. Apart from that, the logistics, with transport to colleges as far as 250km and back, is tremendous. We could save a lot from paper free process! - K.L. Bhatt, COE / Registrar, Bhavnagar University

While funds may not necessarily be a challenge for all universities, the overall examination costs run into billions of rupees. Faster systems, less number of people, paper free exams will all translate into savings and these funds can then be directed towards other more constructive purposes.

6.2.2

Benefits: Increase in reach

Almost all universities and institutes want to reach out to a greater number of students. Many of them use distance education to achieve that. With online exams students find it easier to apply as there are lesser restrictions on travel and time. This in turn results in universities getting and admitting students from across the country.

6.3

Online exams Key concerns

Concerns about the format of exam not being conducive for online came across as the biggest challenge in the survey. However, respondents were not aware about the experiments that are being conducted in universities across the world for better adoption of essay type questions into this system. Similarly

47

infrastructure and lack of technical expertise emerged as concerns too.

Exhibit 23: Infrastructure a major impediment to online exams

Infrastructure Lack of suitability (format) Non-feasibility of multiple batches Technical expertise Budget restrictions Faculty/staff apprehensions Security / reliability Authority Students' resistance Vendor information / selection 0% 20% 40% 60% 80% 100%

Major challenge

Somewhat challenging

Not a challenge

Source: ValueNotes Research

6.3.1

State-of-the-art infrastructure
Lack of infrastructure - Internet penetration in India is still low and many colleges are not equipped with enough number of computers, and existing facilities may not be able to sustain the requirements of an online exam. Support in terms of trained staff is also lacking, and universities have to depend upon external resources for technology support.

Lack of training - Students need more training to be able to give online exams. This is especially true of students residing in rural and semi-urban areas. There are many universities and exam boards that cater to this segment and are not very keen to start online exams in the near future. For instance, Mrs. Chitgeri, Dy. Secretary, MSBTE, said, It is really not possible for us to go online as most of our students are from rural areas. Similarly teachers are not at all equipped to handle problems.
We need to recruit better trained teachers and academic staff. They need to be assessed in their ability to handle computers. - Prof.B.K Kuthiala, Vice Chancellor, Makhanlal Chaturvedi Rashtriya Patrakarita Mahavidyalaya, Bhopal

6.3.2

Resistance to change
Comfort level with the traditional system - Upsetting or changing a system that seems to be working well, is a difficult and controversial decision. The effort starts at trying to change opinions and beliefs of all stakeholders, to proving the efficiency of the new system and implementing the change and then executing the process. The students used to one annual exam at the end, might

48

also resist the change that online exams may bring about that of continuous evaluation. This was cited by a couple of respondents

Decision making While not many stakeholders admit that this is a challenge, we believe that since state and central universities need government approval at every stage especially when it comes to trying something as dramatic as change in technology or assessment, they definitely have to face the typically Indian phenomenon of red tape.

Size of universities - While the size of the universities is the very reason that they should adopt technology, this also becomes the challenge. According to many respondents it is easier for smaller universities to experiment and implement changes, while bigger universities, especially the state and central universities with lakhs of students might find initiating changes more daunting.

Availability of adequate information - There are many companies in India that offer services and products to enable the online environment. However, many respondents said they were not aware of the companies that provided such services and lack of information on best practices, process, adaptability, security and value are major hurdles.

6.3.3

Adaptation of exam format

Scope of online exams - Online exams are ok for those institutions that have to deal with huge numbers and want to mass produce, says Prof. JAK Tareen, VC, Pondicherry University. He vocalized the thoughts of those who say the scope of online exams is limited. Multiple batches was a concept that many were not happy about, especially since it directly impacts the costs. Many others opine that a students ability to express and write is not very effectively graded in online objective format. Allowing students to graduate on the basis of just objective type questions is not fair, according to many decision makers.
Apart from network failures, online exams are heavily biased in favour of language, usually English and that can create problems for those not so competent with it. Also, objective type questions have their limitations about testing a student's knowledge. These are the major drawbacks. - Ms. Viney Kirpal, ex-HoD, IIT-Bombay and CEO of Global Institute of Integrated Training, Pune

Creation of online exams - Most experts who have worked with the online format do agree that creation of exams is the largest and toughest part of going online, as it needs a radical shift in the way we function. Institutes will need to completely understand the outcome of assessment, what are the parameters, how to test those and then come up with question papers.

This then requires research and creation of multiple sets of questions. Students who give these exams in multiple batches often complain of lack of uniformity in question papers.

Limits the examiner - The premise that online exams restrict the format of tests is not very encouraging to stakeholders, who would like to retain their independence and use the format that they prefer. Acceptance of online exams for postgraduate courses, where projects, papers, seminars, etc. form a major part of assessment, is low. Said Prof. Kannan, member, Standing Committee, National Mission on Education through ICT, MHRD, The most difficult thing in my opinion is creation of online exams. We need software technology that can replicate paper-pen exams. This format gives me complete freedom to ask students anything. For instance, an engineering student can be asked to derive a mathematical formula. I am not sure if that is possible in an online exam.

49

Vernacular challenges Indian students speak in a number of languages, and though most higher education institutes are gradually moving to English as a medium of instruction, there are more students who are not comfortable with the language, than those that are. The technology must support multilingual capabilities and this localization involves further expenditures. During our survey, many respondents expressed concern that students from rural areas may face issues.

6.3.4

Unsure about the value of investment

One time or recurring expenditure - While universities understand that adopting online exams may result in just one time expenditure, and over time it may result in reduction in costs, not many are convinced about the long term value. There is a definite shortage of success stories. Hard facts and data on value addition are not adequate. Risks are high and initial costs in infrastructure may be a deterrent, especially for universities with many affiliated colleges in rural locations.
Costs will go up due to investments in infrastructure. We may have to increase fees because entrance exam is a no profit no loss activity. But we are not a commercial enterprise, we cannot charge beyond covering BEP. - Mr. Tripathi, Dy. Registrar (exams), Indian School of Mines, Dhanbad

Considered difficult to implement Another hurdle is that though online exams offer a lot, they do not promise to rectify all challenges faced by the

traditional system. Many respondents said that they tried the online format of exams, and decided not to implement the system.

Opposition to increasing exam fees or budget Most respondents opposed the idea of increasing their budgets to implement online exams.

Exhibit 24: Major reluctance to increasing budgets for online exams

22% 30% Willing to increase budget for online exams Not willing to increase budget for online exams Can't say

48%

Source: ValueNotes Research

Respondents were also reluctant to increase the exam fees charged to students when introducing online exams. Many were of the opinion that the fees should in fact decrease, as online exams offer cost savings.

50

Exhibit 25: Reluctance to increase exam fees for introducing online exams

18% 33% Can increase exam fees for online exams Can't increase exam fees for online exams Maybe Can't say

36% 13%

Source: ValueNotes Research

6.4

Online exams: Execution

This segment deals with the issues related to execution of the entire online exam process. The process involves various stages ranging from registration to actual exams and result declaration.

6.4.1

Execution: Automated administrative tasks

Reduces pressure on administration - Many universities including those that are not comfortable with using online exams are instead initiating online registration and application for students. This has enabled the automation of repetitive and mundane tasks that do not need human intervention. It has also translated into lower manpower needs and lower costs during the process. Most respondents also said they are in favour of moving administrative tasks online.
We are trying to implement automation in some processes, like evaluation. We are also planning retrieval of results by students through their mobile phones. - Mr. V. Rajagopalan, COE, University of Calicut

The following graph also presents an overwhelming majority of respondents who favoured online registration and result declaration. The actual online exam will take some more time to be accepted completely.

51

Exhibit 26: Increasing use of technology in exam process

30% 25% 20% 15% 10% 5% 0% Registration Exam execution Evaluation Result Overall Administrative Others declaration automating tasks systems

Currently using technology

Plan to use technology

Source: ValueNotes Research

Convenience for students Students spend a lot of time in queues to collect forms, paying exam fees and then visiting various universities when they are seeking admissions, or trying to apply for exams. With Internet penetration increasing at the rate of 23% annually, it is hardly surprising that Indian universities are ready to adopt this system.

The Mumbai region plans to train teachers and principals to enable online admissions to junior colleges.

Delhi University also plans to enable online applications for students, again reducing the long queues witnessed every year.

6.4.2

Execution: Establish an efficient system

Faster results and shorter exam cycle With an exam cycle that extends over the entire academic year and results taking as much as 1-3 months to be declared, there is clearly scope for improvement. Respondents also cited the timely declaration of results as one of the biggest challenges. If classroom assessment and other term exams can be attempted, corrected, and tabulated online, at the end of the semester, it is just a matter of generating lists to be collated with the final scores. Nearly 80% of the interviewed university officials mentioned that the ability to declare results on time would have a significant impact on their decision to go online. As Dr. Snehlata Deshmukh, former Vice Chancellor, Mumbai University said, Teachers are expected to
The reason for us to go online are: easy to manage exams, speed, easy for tabulation, early announcement of result, avoidance of human interference and manipulations, etc.. - Dr. S. Sacchidanand, Registrar (Evaluation), Rajiv Gandhi University of Health Sciences

52

check about 30-35 answer sheets every hour. This time is really not enough as there are long answers, and other questions as well. They definitely need more time.

Manpower savings Many universities deploy extra staff during exams. Officials talked about the challenges they face when it comes to bringing in external examiners to complete assessment on time. These examiners stay over a week to ten days to evaluate all answer sheets. Considering the academic cycles of most universities conclude around the same time, the availability and quality of examiners are the two prominent challenges. Automation of objective type of exams will reduce stress on the system.

Better tracking Online exams for classroom evaluation enables better tracking, as faculty and students have access to information on performance at all times. The faculty can thus make informed decisions on the quantum of additional tutoring and other aspects. This enhances the entire learning experience for the students.

6.4.3

Execution: Establish a credible system

Ensuring a transparent system Almost all exams go through numerous rounds of allegations and counter-allegations of wrong doing. A system that will minimize human intervention and be accessible to all offers transparency to stakeholders, which most respondents consider a huge benefit. For instance, with real time evaluation, students are better informed and have higher confidence in the system.
Marks are changed by students with the help pf peons who carry the original answer sheets to their homes where they are re-written by the affected students and replaced by the peons for a consideration. Online exams are good, quick, deliver the result almost immediately and can substitute human intervention of a venial kind. - Ms. Viney Kirpal, ex-HoD, IIT-Bombay and CEO of Global Institute of Integrated Training, Pune

Ensuring a secure process The traditional system relies a lot on human efficiency, and there are parts of process where authorities find it very difficult to maintain control. Transportation of papers back and forth, storing, printing, distribution - chances of mishap exist along almost every step. Users

hope to eliminate the security issues by shortening the cycle and reducing human involvement. Student identification is also considered more fool-proof in an automated environment, as the risk of photo morphing is reduced.

Uniformity in assessment - Automation makes assessment less prone to judgment and other manual errors. Repetitive tasks, process or assessment related are completed efficiently. This will also take care of 'partiality', which is a major concern for students and will reduce over dependence on the human element.
To save time, it is important that universities catering to a very large number of students move online. Similarly we also need to shorten the process of seat allotment to various colleges, as long exam processes and repeated counseling sessions results in the delay of commencement of the academic session. This causes a loss of study time, which needs to be reduced. This, of course, will not be simple, as we need trained staff, need more reforms, adequate infrastructure to move online. - Ms. Neha Pandey, COE, Jagannath University

6.4.4

Execution: User friendly systems

Teachers and authorities Once the staff and teachers receive training on how to handle the systems, they can do it independently, without

53

having to depend on the vendor that has provided them the technology. Adding more questions to the database, generating lists, accessing results, are tasks that can be handled by trained staff. This also enables maintenance of secure systems, as not many people are involved in routine tasks and updates.

Students - In the absence of a common entrance test to graduate and post-graduate professional courses, students in India appear for multiple exams, at times across the country. Though most entrance exams have centres located all over India, the problem of dates clashing and students having to select one of the available options is critical. Online exams facilitate multiple exam dates, enabling students to prepare their own schedule during exam time. They are also able to access provisional certificates and mark sheets enabling them to go for further education.
Exams on a continuous basis is good for regular student as they can attend the classes but for non regular students like working professionals etc the second option is better as they have to adjust with other routine work. - COE at an open university in the North East

Surprisingly, this emerged as one of the lowest priorities during our survey (see Exhibit 22). The graph below illustrates respondents' opinions that they would not be comfortable extending timelines of the exams too much.

Exhibit 27: Preference of time frame for multiple batches

24% 18%

24% 21% 15%

1 day

1-3 days

4-7 days

7-15 days

A month

Source: ValueNotes Research

About 65% of respondents said that though they are open to conducting exams over multiple days, they would not want it to go beyond a week. Some respondents also mentioned that multiple batches, with students being able to choose time slots would not be feasible. There would be challenges in co-ordination and greater security risks for the exam department.

6.5

Conclusion

The findings of our survey have shown that exam authorities need to take a hard look at exam processes if/when they are considering transitioning to online assessment. However this is not to say that the

54

challenges discussed cannot be overcome. What we need is continuous flow of information, and sharing of knowledge and experience. CAT 2009 exposed the failures, but it also proved that online exams are possible in India (please refer to Annexure 2 and Annexure 3 for more analysis on the feasibility of online exams in India). What is needed is careful scrutiny, better preparedness, more training and ability and willingness to drive change. The stakeholders in the education sector administrators and students are still getting used to the idea of taking exams online. While those that have experienced the benefits of ICT in education are enthused, many others are willing to give it a try. Yet there is definitely a wait and watch approach. CAT 2009 was definitive in a way, as many put their plan to go online on hold, based on the somewhat disappointing results. It will take CAT 2010 to be successful, and other smaller success stories to come out in the open and the number of those willing to experiment will increase manifold. As education expert Rahul Counhaha states, The CAT fiasco, although significant, is a short-term phenomenon. Service providers and institutions will learn from these examples to improve the testing process and delivery. Likewise, online assessment is going to become more sophisticated as the technology applications improve.
Gradually Indian higher education should move towards online. Once we start, improvements will follow. We also need to move towards continuous assessment, and more flexibility in exams. - Ms Uma Kanjilal, an education expert - Prof Kannan, Member, Standing Committee, National Mission on Education through ICT, MHRD The experience with CAT entrance this year is not likely to be repeated, I don't think there will be a problem again.

55

7.

Student Speak

Students are the most important part of the education system, and the exercise of taking exams online will yield many benefits to them. Yet Vice chancellors we spoke to mentioned that there was a certain amount of reservation expressed by students and therefore adoption was slow. In this chapter, we will present views and opinions of one of the most important stakeholders of higher education examinations. Even as efforts are underway to make ICT an inherent part of our education system, are the students really ready?

7.1

Profile of respondents

Towards bridging the perception gap of university officials and students, ValueNotes conducted a nationwide survey of 400 students from across Tier I, Tier II and Tier III cities, studying mostly at graduate and postgraduate levels, with a few inputs from junior college and PhD students. The aim was to capture the students' awareness, readiness and experiences with online examinations.

7.1.1

General information

Exhibit 28: Age brackets and gender distribution of respondents

Age brackets in years

Gender

15%

2%

37%

Female 40% Male 60%

46%

17-20

21-23

24-26

27-30
Source: ValueNotes Research

Majority of students are aged between 17-23 years, with 40:60 female:male ratio.

56

Exhibit 29: Distribution of respondents' academic level and city of study

Academic level

City of study 34% 30%

33%

36%

1% 1%

66%

Junior college

PhD

Postgraduate

Undergraduate

Tier I

Tier II

Tier III

Source: ValueNotes Research

Majority of respondents are at the undergraduate level, followed by postgraduate students. ValueNotes interviewed an almost equal distribution of respondents studying in Tier I, Tier II and Tier III cities. The survey has not included the opinions of students from rural areas. The list of cities is as follows: Ahmedabad, Bangalore, Bhopal, Bhubaneshwar, Chandigarh, Chennai, Coimbatore Delhi and NCR region, Hyderabad and Secunderabad, Indore, Jabalpur, Jaipur, Kanpur, Kolkata, Kota, Kozhikode, Lucknow, Mangalore, Mumbai, Nagpur, Nashik, Pune and Salem. Exhibit 30: Distribution of respondents' stream of study

Stream of study
4% 1%

8%

19%

7% 27% 1%

34%

Bachelor's Engineering Medicine

Computer science Management PhD

Source: ValueNotes Research

Diploma Master's

A large part of the sample consists of engineering and management students.

57

7.1.2

Ownership of technology

Exhibit 31: Ownership of computer/laptop and Internet connection

64% 55% 45%

Ow n a computer

Ow n a laptop

Ow n an internet connection

Source: ValueNotes Research

Findings

83% of respondents possessed a computer or a laptop. Surprisingly, many students were reluctant to admit that they did not possess an Internet connection, stating that they would soon procure one.

Exhibit 32: High frequency of usage of technology

42%
> 10 hours a w eek

56% 20% 20% 22%

6-10 hours a w eek

1-5 hours a w eek

18% 9% 6%

<1 hour a w eek

Usage of computer / laptop

Usage of internet

Source: ValueNotes Research

Findings

There is a strong correlation between computer / laptop usage and the usage of the Internet. The variance is for students who use their computer / laptop for more than 10 hours a week, but not the Internet, for activities such as typing assignments.

58

7.1.3

Academic use of technology

Exhibit 33: Similar academic use of technology across city tiers

100%

80%

60%

40%

20%

Typing assignments/projects

Communicating with students/faculty

Applying for exams/courses

Applying/registering for other purposes

Surfing for course related material

Tier I

Tier II

Tier III

Source: ValueNotes Research

Findings

Students make use of technology for varied academic purposes, more so for browsing the Internet for course related information.

Students across city Tiers use technology for academic purposes in a similar manner. However, more students from Tier III cities use the Internet to surf for study material, perhaps signalling a lack of quality curriculum in these cities.

Students across undergraduate and postgraduate courses use computers and the Internet for similar academic purposes, in similar ratios.

Apart from the purposes listed above, students also use computers and the Internet to give mock exams, research universities/colleges, take e-tutorials, research online libraries and e-journals, and contribute to student forums (blogs, wikis, discussion boards, networks).

Others

0%

59

7.2

Awareness and experience with online exams

7.2.1

Most students are aware of online exams

Exhibit 34: Most students are aware of online exams

Unaware of online exams, 5%

Aware of online exams, 95%

Source: ValueNotes Research

The overwhelming majority of students (95%) are aware that institutions conduct exams online. Exhibit 35: Students originating from Tier III cities are less aware of online exams
100%

80%

60%

40%

20%

0%

Aware of online exams Tier I Tier II

Unaware of online exams Tier III

Source: ValueNotes Research

65% of students who aren't aware of online exams originated from Tier III cities

60

Exhibit 36: Students who are not aware of online exams do not possess an Internet connection
100%

80%

60%

40%

20%

0%

Aware of online exams Don't own an internet connection

Unaware of online exams Own an internet connection

Source: ValueNotes Research

85% of students who aren't aware of online exams did not possess an Internet connection

7.2.2

Majority of students have no experience with online exams

Exhibit 37: Majority of students have no experience with online exams

7% 10%

55% 28%

No, never used Yes for college exams

Yes for competitive exams Yes for both

Source: ValueNotes Research

61

Exhibit 38: Tier I students have more experience with online exams

100% 80% 60% 40% 20% 0%

No, never used Yes for both

Yes for college exams

Yes for competitive exams

It actually depends on the type of examination being conducted online. MCQ exams are quite easy, I think

Tier I

Tier II

Tier III

- Zafar, Industrial design, Nashik

Source: ValueNotes Research

Findings

Students in Tier I cities have more experience with online university exams. Further, younger students have not given too many online exams; universities are experimenting with more online PG entrance exams.

Vast majority of students (71%) who have given both entrance and university exams online are studying in Tier I cities, along with the majority of students (60%) who have given university exams online.

Majority of respondents (40%) who have never given an online exam are studying in Tier III cities. Tier III students have more experience with competitive exams

62

7.3

Readiness towards online exams

7.3.1

Conducting online exams sends positive messages to students

Exhibit 39: Conducting online exams sends positive messages to students

More technically advanced, progressive university Similar to other universities

69%

17%

No opinion

14%

Source: ValueNotes Research

The vast majority of students feel that institutions which conduct online exams are more technically advanced and progressive.

Exhibit 40: Students originating from Tier III cities far more impressed with online exams
100% 80% 60% 40% 20% 0%

Tier I

Tier II

Tier III

Similar to other universities No opinion more technically advanced (and hence a progressive college)
Source: ValueNotes Research

If the university has the capability to conduct its exams online, then it must definitely be more technically advanced. - Ashish, B.Tech, Lucknow

Findings

More students originating from Tier III cities feel that institutions which conduct online tests are more

63

technically advanced or progressive.

Vast majority of students originating from Tier III cities (~80%) claimed universities that conduct online tests are more technically advanced. This is in contrast to Tier I (56%) and Tier II students (61%).

More students from Tier I and Tier II cities feel universities that conduct online tests are similar to others. Over 60% of respondents who have no opinion on the subject are studying in Tier I cities.

7.3.2

Online exams score high on security and credibility

Exhibit 41: Online exams score high on security and credibility

Online exam process is more secure and credible

46% 16% 23% 15%

No difference perceived

Its exam process is less secure and less credible

Not sure/No comment

Source: ValueNotes Research

Findings

Majority of students feel the online exam process is more secure and credible.

Online exams like CAT have had more problems due to security, I would prefer not to give them online. - Vivek, MBA, Bhopal The exam process is more secure online, but it totally depends on the load on the server, i.e. number of candidates vs. the server capability. - Vaibhav, B.Tech, Bhopal

There is also a significant percentage (23%) which considers the security and credibility of online exams to be less than traditional exams.

64

Exhibit 42: Experienced students perceive no difference in security

70% 60% 50% 40% 30% 20% 10% 0% Online exam process is Online exam process is less secure and less more secure and credible credible No, never used Yes for both No difference perceived Not sure / No comment

Yes for college exams

Yes for competitive exams

Source: ValueNotes Research

Majority of students (~40%) who have given both entrance and university exams online believe there is no difference perceived in the security and credibility of process.

7.3.3

Students feel online exams are far more accurate

Exhibit 43: Students feel online exams are far more accurate

Online exam process is more accurate and authentic No diff erence perceived

68%

15%

There is low er accuracy / authenticity of the results

7% The accuracy of online exams is extremely high, it is in fact perfect. - Siva, B.Tech, Salem
Source: ValueNotes Research

Not sure/No comment

9%

65

Findings

Vast majority of students feel that the online exam process is more accurate and authentic, compared to traditional exam system.

There is also a significant correlation between the percentage of respondents who consider the security and credibility of online exams to be higher/lower/similar/unsure and the accuracy and authenticity of online exams as higher/lower/similar/unsure, respectively.

Exhibit 44: Speculation among students who haven't given online exams

100% 80% 60% 40% 20% 0% Online exam process is more accurate and authentic No difference perceived There is low er accuracy / authenticity of the results Not sure / No comment

No, never used

Yes for both

Yes for college exams

Source: ValueNotes Research

Yes for competitive exams

Findings

Due to issues with recent national level online exams, there is speculation with respect to accuracy and authenticity among students who have not yet given online exams.
Accuracy of results depends on the type of exam being taken. Objective exams would have 100% accuracy, but how accurately can a computer assess a subjective paper? - Rashid, MCA, Lucknow

The majority of respondents who believe there is lower accuracy / authenticity (67%), or are not sure (80%), have never given an online exam

66

Exhibit 45: Students who feel accuracy is higher are more likely to join a university which conducts online exams

70% 60% 50% 40% 30% 20% 10% 0% Online exam process is more accurate and authentic No difference perceived There is low er accuracy/authenticity of the results Not sure/No comment

Less likely

More likely

No opinion

No preference

Source: ValueNotes Research

The majority of students (50%) who believe the online exam process is more accurate, are more likely to join a university which does online testing. Similarly, the majority (36%) who consider the process to be less accurate, are less likely to join a university which does online testing.
The accuracy is great, you get immediate results too. - Priya, B.Com, Bhopal

7.3.4

Students have preferences in type of exam to take online

Exhibit 46: Students have preferences in type of exam to take online

26%

74% No preference in type of exam to give online Some preferences in type of exam to give online
Source: ValueNotes Research

67

Findings Majority of students did have an idea of the type of exams they would prefer taking online. The consensus was on the following:

Objective tests of any kind Class tests and mid-terms (in-course exams) Entrance exams, especially for MBA, engineering, UPSC and IAS Exams in different locations, such as foreign university exams Exams for professional certificate courses (such as Java) Non-academic exams (driver's license, passport, etc.)

7.3.5

Students more likely to join a university which conducts online exams

Exhibit 47: Students more likely to join a university which conducts online exams

More likely

42%

No preference

35%

Less likely

15%

No opinion

8%

Source: ValueNotes Research

Findings

Many students do not have any preference in exam format, when picking potential universities. Respondents mentioned it is not a critical factor, and they would instead focus more on the overall reputation of the university before joining.
I would prefer paper and pencil exams, because some online tests don't allow me to edit once selected answers. - Kumar Gaurav, MBA, Indore

Half the respondents who have given an online university exam claim to be more likely to join a university which conducts an online exam

81% of those more likely to join a university which conducts an online exam also believe that it would be more technically advanced, and hence a progressive university.

Yes, I am more likely to join, especially if the university's internal assessments are online. - Sonam, PGDM, Bangalore

68

7.4

Perceived benefits and key concerns

7.4.1

Students see many important benefits of online exams

Exhibit 48: Perceived / expected benefits of online exams

100%

80%

60%

40%

20%

0% Convenience in applying Ease of taking exam Greater efficiency Higher confidence Transparency of Accurate and processes timely

Major benefit

Somewhat of a benefit
Source: ValueNotes Research

Not a benefit

Findings

Students see similar benefits with being able to give entrance and university exams online. However, they do not yet have a greater level of confidence with the online exam process, compared to traditional exams.

If the site is user-friendly and straight forward, it would be very convenient to apply online instead of running circles around colleges trying to get all the documents together. - Prasanjeet, MBA, Pune Online exams which display the consolidated results immediately are great. They eliminate the pressure and anxiety between exams and result declaration! - Sagar, MBBS, Pune

The convenience in online application/registration is greater by 32% for competitive exams.

69

7.4.2

Support and difficulty of usage the key concerns

Exhibit 49: Perceived / expected issues with online exams

100% 80% 60% 40% 20% 0% Security risks Infrastructure Difficult to use Support

Major issue

Somewhat of an issue
Source: ValueNotes Research

Not an issue

Findings

Students perceive similar issues with online exams at entrance and semester levels. Support is seen as a major concern, especially given the current speculation around online exams.

This is still the initial stage and initially there are always problems... but it will smoothen out soon. - Aditya, BCA, Jaipur Students should take up computer courses and use computers recreationally so that there is no unfamiliarity. - Sandeep, MBA, Bhopal The infrastructure in India needs to improve. The recent CAT and similar exams, which were not conducted properly have the potential to spoil one's career. - Deepak, B.E, Jabalpur

Infrastructure is seen as more of a major concern for competitive exams (62%), compared to university exams (34%).

The majority of respondents (43%) who feel infrastructure is not a concern for online university exams are studying in Tier I cities.

Further, the majority of students (39%) who claimed infrastructure is a major concern for online university exams are studying in Tier III cities.

The majority of students (~53%) who feel that difficulty of usage is not an issue (for competitive / university exam) have had some form of formal training on the use of computers, intranet / Internet or software.

7.5

Conclusion
The survey revealed that an overwhelming majority of students (95%) are aware of online assessments.

Majority of students (69%) felt that these universities are more technically advanced, though students originating from Tier III cities are far more impressed.

70

Majority of students (46%) consider the online process to be more secure, compared to traditional methods.

The accuracy and authenticity of online exams is far higher, according to majority of students (68%). There was a strong correlation drawn between these students, being more likely to join a university which conducts online exams.

More than half the students (55%) interviewed had no experience with online exams, at the university or entrance level.

Transparency of processes is seen as the major benefit, followed by greater efficiency. The convenience in online application/registration is perceived to be greater by 32% for competitive exams.

Students are most concerned about the difficulty of usage of online exams, and lack of support from exam authorities.

71

8.

Conclusion

Few can deny that the huge demographic dividend that India enjoys (in terms of % of young people) poses both a great challenge and an equally great opportunity. If India can educate all her youth, then jobs and prosperity will follow, especially given the aging populations of the developed countries. If we fail on education, however, the consequences will be continued poverty and social unrest. Fortunately, this issue has now become a mainstream topic for debate. While most discussions and reports focus on the need to improve curriculum and teaching (which are no doubt, extremely important), few talk about the need for measuring the learning outcomes, or assessment. This report, is the first that looks at the challenges of assessment, and the potential benefits of technology in the country. We have also tried to assess the awareness, perception, readiness and experience of key stakeholders in Indian higher education with respect to technology solutions to assessment, primarily online examinations. This was based on an exhaustive, countrywide survey, where we spoke with more than 75 decision makers (vice chancellors, exam controllers, university registrars, experts, etc). The survey findings, to say the least, are very interesting:

Several universities are experimenting with online assessment systems. Decision makers largely believe that technology solutions can mitigate some of the immense challenges we face.

The level of awareness of various exam authorities about potential technology solutions for assessment are quite high.

An extremely encouraging ~50% of the respondents stated a positive intent to move exams online in the next 6-36 months.

The top three priorities in exam management were faster results, greater transparency, and student authentication. Respondents said these would have the most impact on their decision to go online.

Infrastructure and lack of suitability emerged as the top two perceived challenges with almost 60% of the respondents referring to them.

Some respondents were also worried about the students from rural areas, who may not have access to computers at all, and may face difficulty in adopting this.

Our estimates suggest that the number of exams in India annually, number over 200 million. We believe that if the exams in India were to move online, the current cost of Rs. 35,300 52,400 million will come down significantly.

Our respondents don't agree on the quantum of achievable cost savings due to technology, as the range of opinion varies from 25% to as high as 80% (savings).

The Edge ValueNotes' study also included a student survey, to assess how students feel about this

72

issue. The survey was conducted in 20 cities which that included Tier I, Tier II and Tier III cities.

An overwhelming majority of students (95%) are aware of online assessments, and this is consistent even in Tier III cities.

The majority of students (69%) felt that universities that conduct online exams are more technically advanced, though students originating from Tier III cities are far more impressed.

A significant proportion of students did consider the online process to be more secure, compared to traditional methods.

The accuracy and authenticity of online exams is far higher, according to most students (68%). Transparency of processes is seen as the major benefit, followed by greater efficiency. The convenience in online application/registration is perceived to be greater by 32% for competitive exams.

Interestingly, more than half the students (55%) interviewed had absolutely no experience giving exams online, at the university or entrance level.

Students are most concerned about the difficulty of usage of online exams, and lack of support from exam authorities.

In conclusion, the results of this study point to some very encouraging trends. Firstly, decision makers are not only aware of the benefits of using technology for assessments, they have indicate positive intent as regards implementation of online assessment. Despite the IIM (CAT) problems, universities still believe in the positive impact, even if they are more focused on correct implementation. In fact, the early shock may be a good thing! Students too are positively inclined, despite concerns being raised by sceptics on their behalf. Interestingly, the less technology-exposed students (in Tier III cities) are the most positive. Given all the factors above, it appears that online exams in India are set for a fairly long period of strong growth. No doubt there are many challenges, especially security and reliability, but the trend is clear.

73

9.

About ValueNotes

ValueNotes is a leading provider of business intelligence and research, with expertise across industries, particularly in financial services, media, engineering, healthcare, IT and the outsourcing industry. Our strengths lie in the strong analytical ability, research and data collection skills of our researchers and analysts. We work with clients across the globe, and we have significant understanding of international markets. We provide a wide range of bespoke business and financial research services about specific markets, industries, companies and competitive environments. We serve leading global corporations, consulting firms, research and B2B publishers, PE and VC firms, and money managers. ValueNotes.com: The website provides for an independent and unbiased aggregation of views, opinions, research, analysis and insights on the Indian financial markets. We seek to understand, inform and educate the investor, and partner with banks, asset managers, stockbrokers, insurers, merchant bankers, and independent financial analysts. ValueNotes Sourcing Practice: We publish proprietary market intelligence on the (services) outsourcing industry - in BFSI, e-learning, healthcare, market research, legal, publishing, and engineering - with an emphasis on knowledge services or KPO. We are now one of the largest information providers on the outsourcing industry. Our subscribers include buyers of outsourcing services, providers, consultants and investors. www.SourcingNotes.com is the official website of the Practice, intended to showcase our research, products and publications. Research Publications Some of our recent published research on the outsourcing industry include:
ValueNotes Sourcing Prism: Legal outsourcing edition Legal What do STM / academic publishers think about outsourcing? Process Outsourcing: Crisis creates new opportunities Offshoring in the publishing vertical - 2009 Legal Services Outsourcing: What do law firms think? Market Research Outsourcing - Buyer survey e-learning Outsourcing 2009: Advantage India Outsourcing in Malaysia: Scaling new heights

Custom Research Some of our recent custom research projects include:

Opportunity assessment
Customer analysis and distributor search for a US Partner

Partner selection

identification for a US manufacturer of professional knives manufacturer aerospace products Market study for a German utilities company to Identification and evaluation of potential partners in evaluate the utility metering market potential in India India to source diamond and gold jewelry for a French retail chain Opportunity assessment for whey products in India for a global consulting firm Identification of vocational institutes in India as potential partners for a Dutch university Set up a captive offshore operation for a financial services firm Rice exporter identification study for a major European consulting firm Opportunities for venture capital and private equity investment in Indias IT sector
Industry overview and partner due diligence in medical

transcription

74

Competitive intelligence publishing industry

Benchmarking for an engineering design outsourcing

Market intelligence
Human resources manual for a financial services

Evaluation and benchmarking of vendors in the legal Cost analysis for a legal outsourcing vendor

captive BPO
Designed, constructed and structured the first-ever

operation
Profiled beer manufacturers in select Asian countries

investor and advisor focused Investment Confidence for a UK consulting firm Index in India for a global fund manager Product launch intelligence for a US pharma company Analyzed the usage of T&E corporate cards by to assess the therapeutic potential of a psycho-active companies in India, Australia, UK, South Africa and substance in the Indian and Chinese markets Ireland
Investigated channel partner programs of handheld

device providers for a reputed consulting firm

To get in touch with ValueNotes, please call us on (+91 20) 6623 1743 / 1717 or email us at [email protected]

75

10. About EDGE Forum

EDGE Forum Emerging Directions in Global Education is a collective initiative first launched in 2007 by leading educational institutions and organizations from India and abroad. EDGE Conference brings policy makers, heads of educational Institutions and industry on one platform to address issues pertaining to excellence in higher education. This initiative aims at bringing together global educationists to forge cooperative and collaborative programs to further the knowledge exchange and internationalise education. Highlights of EDGE 2010:

Focal theme: Catalyzing Change Inauguration in the presence of dignitaries Shri Kapil Sibal, Honble Minister for HRD, Government of India, Dr. Martha Kanter, Under Secretary, Education Department, USA, Mr. Sam Pitroda, Advisor to Prime Minister of India, Dr. K Kasturirangan, Member, Planning Commission, Dr. V.N. Rajasekaran Pillai, VC, IGNOU and Prof V.S. Ramamurthy, Director, NIAS.

Participation of over 350 delegates consisting of top Educators and Policy makers and a delegation of top University Presidents led by Institute of International Education.

Education Personality of the year 2010 - Prof. Yashpal. EduPartnering: A platform for delegates from international universities to meet educators from India for exploring tie-ups and collaborations.

EDU Expo: Showcase of different streams in education, technologies & services. Release of five Exclusive Research Reports NIAS - EDGE Report on Catalysing Change FED UNI - Parthenon Report on Bridging The Higher Education Gap: National Universities And Other Models

ValueNotes - EDGE Report on Examinations and the Role of Technology: Emerging Directions EDGE - Ernst & Young Report 'Building Blocks for Education' Institute of International Education (IIE) Report on International India: A Turning Point in Educational Exchange with the US Conference Topics:

Role of Technology in Improving Effectiveness & Efficiency of Examination Process Catalyzing Change: How International Education is Transforming our Campuses Quality Improvement Initiatives in Engineering Education Engaging India: Initiatives at US universities-conducted by AIEA ICT Empowering Education Progressive Directions in Online Education: for Quality & Reach Vocational Education by Choice Workshops: 'Lessons Learned: Issues to Consider in Building a National Admissions Test' by The College Board, USA

'Education Governance' by a Panel of Management Experts

76

11. About MeritTrac

MeritTrac is India's Largest Testing & Skills Assessment Company. MeritTrac provides innovative examination process and assessment solutions to Educational Institutions, Government/ PSU sector and leading corporate entities. MeritTrac will deliver around 3.5 million exams in FY2009-10 and has delivered close to 8 million exams till date - both in a paper-pencil mode and online mode. Headquartered in Bangalore, and having presence in 16 cities in India, including a network of company-owned testing centers, MeritTrac has over 300 full-time employees. The test research & test development teams have created a battery of over 300 validated tests, spanning areas like communication skills, general abilities, domain and behavioral skills. The test delivery teams have delivered tests in over 250 cities in the country with the distinction of having delivered 1 million exams, in a period of 10 days across 175 cities. MeritTrac has created Indias largest test center network of 50,000 terminals with authorized Pariksha Centers being activated in 185 cities in India. MeritTrac has also defined robust processes for examinations which are ISO 9001:2008 certified. In the Education sector, MeritTrac works with prestigious institutions like Gujarat Technological University, JIPMER Pondicherry, IIIT-Bangalore, Manipal University, Sikkim Manipal University, NMIMS University, NIMHANS and Indian Institute of Banking & Finance. Some of its clients in the Government/PSU include Govt. of Orissa, Govt. of Gujarat and over 12 large Public Sector Units. HP, Microsoft, Accenture, Cognizant, Wipro, ICICI Bank, Titan, Axis Bank, Jindal Steel are some of its prestigious clients in the Corporate sector. MeritTrac is also the exclusive assessment partner to NASSCOM for its NAC-Tech certification Program and is an accredited Assessment body for the Modular Employable Skills (MES) program by DGET, Ministry of Labour & Employment. MeritTrac has been an acknowledged leader and pioneer in the field of testing and has been bestowed with several awards & recognitions like the eIndia ICT Award & the Manthan South Asia Awards 2009 for the online Gujarat Common Entrance Test (GCET) project - awarded jointly with Gujarat Technological University, and NASSCOM IT Innovation Award 2007. MeritTrac is also a full-member of Association of Test Publishers (ATP), US which is one of the prestigious associations of test publishing companies - the first company from India to be admitted to this association, and an active member of NASSCOM, CII and FICCI-Higher Education Network. Contact: Corporate Office: MeritTrac Services Pvt. Ltd 2F,GK Arcade, T.Mariappa Road, Jayanagar 1st Block Bangalore - 11, INDIA Tel (Board): +91.80.40130200 Fax: +91.80.40130208 Email: [email protected] Visit us: www.merittrac.com

77

12. About Excelsoft

Excelsoft is a provider of innovative technology-based solutions in the education and training space. We architect, design and develop large technology solutions and digital content for some of the best known organizations, worldwide. SARAS Our e-learning technology solutions framework and the product suite developed around it has successfully penetrated the international markets in corporations, universities and schools, publishers, elearning portals and governments. SARAS enables transformation of businesses into learning organizations, enhances the effectiveness of teaching-learning-assessment processes in educational institutions and provides newer revenue streams for publishers. The key attributes of SARAS are attractive Total Cost of Ownership (TCO), quick Return on Investment (RoI) and rapid deployment. SARAS architecture conforms to Service Oriented Architecture (SOA) governance standards and enables Reusability, Extensibility, Scalability and Performance. Our e-learning content development practice is a process-driven model that blends strong educational pedagogy with good instructional design. We have the experience to develop content for a wide spectrum of clients from universities, colleges and schools to corporate houses. Our global exposure helps us deliver cost-effective, professionally developed content. Experienced instructional design and course authoring teams are our bulwarks in developing quality e-learning content. We specialize in enabling setting up of realistic success metrics. This ensures accurate gauging of learners skill levels during and upon completion of the course. Our close association with the education sector in India and abroad provides us access to a large number of subject experts who form an integral part of our content development team. We take into account the nature of the target audience, its present level of education, its areas of interest and the best method by which the interest of the learner in the subject can be aroused and sustained throughout the material designed, with assessment, so that REAL learning takes place. With over 1000 man-years of experience in content development, we have so far created highly engaging, interactive content in Higher Ed, K-12 and corporate space amounting to over 5000 learning hours. Our clientele includes global leaders in educational publishing such as Pearson Education, Harvard Business school of Publishing, Cengage Learning, Infinitas Learning, the Guardian Media Group and others, some of the worlds best known universities such as Oxford and Cambridge, Anna University & Visvesvaraya Technological University in India, a host of middle east universities, prestigious corporations such as Infosys, T.I.M.E, ICICI Bank, Bajaj Allianz and more. Excelsoft is headquartered in Mysore, with offices in Bangalore, Noida & Hyderabad. Our International sales offices are located in Singapore, USA & UK. To know more about us please visit www.excelindia.com

78

Annexure 1: List of participating universities and exam boards

Names of universities / exam boards Rajasthan Technical University Sastra University Gokhale Institute of Politics and Economics Vignan University Yashwantrao Chavan Maharashtra Open University Visva Bharati University Tilak Maharashtra Vidyapeeth The Indian Institute of Foreign Trade (IIFT), Delhi IMT, Ghaziabad Jawaharlal Nehru University K. K. Handique State University Sri Chandrasekhara Saraswati Vishwa Mahavidyalay National Institute of Technology, Kurukshetra KIMS University Ravenshaw University Avinashilingam University for Women Punjab Technical University Dev Sanskriti Vishwavidyalaya Jagannath University Sri Ramaswamy Memorial (SRM) University Kalyani University Behrampur University University of Mysore Maharashtra University of Health Sciences (MUHS) University of Calcutta Tata Institute of Social Sciences Krishna University Rani Durgavati Vishwavidyalaya Indian Institute of Management, Indore Sri Chandrasekhara Saraswati Vishwa Mahavidyalay Vels Institute of Science, Technology and Advanced Studies (VISTAS) Kannada University Makhanlal Chaturvedi Rashtriya Patrakarita Mahavidyalaya Jai Narayan Vyas University Indira Gandhi National Open University (IGNOU) Indian Institute of Technology, Bombay University of Mumbai Maharashtra State Board of Technical Education Madhya Pradesh Professional Examination Board (VYAPAM) Jawaharlal Institute of Post Graduate Medical Education and Research (JIPMER) Karnataka Examination Authority (KEA) University of Pondicherry Tolani Maritime Institute Indian School of Mines, Dhanbad Karnataka State Open University Rajiv Gandhi University of Health Sciences Chaudhary Devi Lal University University of Calicut Cochin University of Science and Technology Pt. Ravishankar Shukla University Veer Narmad South Gujarat University Kannur University Bhavnagar University Jadavpur University Netaji Subhas Open University Ganpat University

79

Annexure 2: Delivering online exams consistently I have headed a team that has delivered over a million computer-based exams in India over the past few years. Through this experience, we have learnt a few things about delivering online exams which I describe below. Online exams are a great example of technology and trained people coming together to deliver a great experience to candidates. And these exams are as much about trained people as they are about technology. The technology for a computer-based exam consists of an examination software application installed on a server. The examination software platform has back-office and front-office modules. The back-office modules manage hall tickets/attendance sheet creation, question paper generation/distribution and report publishing processes where there is no face-to-face interaction contact with the candidate. The frontoffice modules manage processes where there is face-to-face interaction with the candidate. Back-office modules The back-office modules of the examination software application reside in a single location and are run off a few servers in a highly-controlled environment. A small number of people manage the back-office and it is easy to ensure complete compliance with processes. Ensuring that the back-office technology is up and running, switching to failovers in case of a problem and ensuring that data on the back-office modules is secure is simple. But the front-office modules of the application present an array of challenges owing to the fact that they have to run on thousands of PCs across hundreds of exam centres. The front-office technology consists of checking-in candidates and presenting examinations to candidates. Checking-in candidates in an examination happening at a 100 exam centres across the country would mean each check-in desk would be required to check-in about 50 candidates (assuming 5,000 candidates are taking an exam at one time.). And all 5,000 candidates have to be checked-in in an hour (since candidates would report an hour before the exam). This is similar to checking-in passengers at an airport across a 100 check-in desks. The difference is that every passenger's biometric and photograph has to be recorded at check-in desks spread across different exam centres. The challenges in doing this is that check-in desks have to be fast since examinations at different centres cannot start at different times and all check-in desks have to complete checking-in all the candidates in time. Owing to this, the check-in technology has to work fast quickly capture a candidate's biometric and photo, check details and admit the candidate. In addition, the check-in technology has to have a built-in database, OS and hardware redundancy which means the technology has to be deployed on two different servers and the check-in data on the server has to be automatically copied to the other server. In case one server fails, the back-up server is used to do the check-in. Presenting exams Presenting examinations to candidates is the most important part of an examination. This is where the rubber hits the road. As a first step, the user interface design needs to be discussed, debated, tried out, modified, tried out again and then finalised. Every little thing on the computer screen matters. It has to be debated whether a button should be named End Test or Quit; whether the End Test button should

80

come up only when the candidate is on the last question; whether navigation buttons that allow the candidate to move to any question should be on the left hand side or at the bottom of the screen; the font size of the questions that come up; the area where images are shown; will candidates require to horizontally scroll; or will they have to vertically scroll. The next step is to put the user interface on a Web site and have candidates try it out and provide feedback. The feedback needs to be incorporated into the final release. The idea is to give every candidate an opportunity to try out the user interface several times before the actual exam. This is the only way fairness can be achieved. The third step is ensure that the exam presentation technology has extraordinary exam resume capabilities. This would mean that the technology should allow resuming the exam from any point. And the technology has to be stress-tested for several power cuts in a short time span (a situation unique to India); pulling out a network cable; a keyboard not responding; the OS hanging; and so on. The exam presentation technology has to assume that exams will be interrupted. It has to be able to restart where it left off, no matter what. And it has to restart with no disadvantage to the candidate. In India, computer labs in colleges and other locations are not set up as best-in-class infrastructure because it does not make commercial sense to the institutions. Also, to expect that 5,000 computers will work correctly at a given point in time is inherently poor planning. So, it is imperative that the exam presentation technology instead is best-in-class so that it runs on virtually any quality of infrastructure. While technology is crucial, the quality, training and response of the people who meet and administer the exam is what finally matters. This would mean the people who are deployed to do the exam are trained to ensure processes are completed efficiently. Also they should be trained to ensure any disruption is handled effectively with minimum disturbance to the candidate. Thus training should cover various scenarios when a check-in desk goes down or a biometric device stops working or the computer the student is working on hangs up.. Finally, an examination project is only as good as the project management that goes into planning the exam. The project management has to look into all aspects of the exam communication with students; helpdesk; applications; schedule; question papers; exam delivery; etc. Like all good projects, there should be identified risks and known and planned mitigation steps. The planning should take into account a range of uncertainties proctors reaching exam centres late, flights getting delayed / cancelled, trains reaching late, power outages, and the like. Perhaps the most crucial thing is to ensure that the project management team is experienced in managing a computer exam in the country. Delivering computer-based exams brings together people, processes, infrastructure and technology. With a rigorous project management approach, it is possible to deliver online examinations successfully. - Mohan Kannegal (This article appeared in The Hindu, dated 07 Dec 2009. The writer is Co-Founder and Chief Delivery Officer, MeritTrac Services.)

81

Annexure 3: CAT in mousetrap: A failure of management, says expert The technical snag which hit the online CAT has more to do with project management skills than anything else. "It could not be taken as the failure of online testing," said Upendra Kumar, managing director, UPTEC Computer Consultancy Limited. Moreover, an audit by the third neutral party into the whole thing can bring out the anomalies that led to the `mismanaged' first-time online CAT, he said, adding "audit will help bring out the facts. And if the need be, or so say the audit findings, the paper and pencil test might be brought back to restore confidence in CAT system." "There were no dry runs of the system and no quality assurance checks," said Kumar. The system could have been tested over mock papers with dummy candidates appearing in them. The process could have led to lot of snags been removed then and there. The whole thing could have been taken up at the level of a project with a project head supervising every step of the system. Kumar, who is is former IIM-Lucknow professor in IT, said "Project head could have monitored the efforts down the line." Explaining the technicalities of online testing, he said that there are specific components of it. First is the System, which includes architecture, process and software. The IIMs had contracted Prometric to hold the online CAT. "There is a need for someone to check if the architecture and process used in other online tests (for lesser number of candidates) has been used in online CAT as well," said Kumar. The biggest fault with CAT online could have been the large number of candidates tested over a short period of time. The very first attempt to hold an online CAT has faced process delays, non-working of biometric systems, passwords that did not work or were common, repeated and non-visible questions and repeated answer options. In some of the cases, clicking on `quick review' has also closed down the test. The problems like repeated questions could have something to do with the size of question bank. "Again an audit can find out if the size of question bank was small and what was the quality of questions," he said. Coming to `delivery', which comprises the third component of online testing, there has to be sanitisation of examinations lab and equipment. "There is a need to completely isolate the lab where online test is to be held," he said. Here, mostly college computers used by students over several years were put to use. The `delivery' also calls for proper proctoring, management and administration. "A proper training of the proctors is to be ensured so that they can help the candidates," said Kumar. Another important missing factor here could be the lack of fallback options. "There has to be excess capacity so that if one computer crashes or there is a problem at one centre, candidates can be comfortably accommodated," he added. The technical glitches have certainly left candidates who appeared for the online CAT wondering if their answers had been captured correctly or how will all this lead to correct evaluation of his/her marks. "Since CAT is a high profile test and has high-visibility, an audit by third party can come out with exact reasons which led candidates face problems," said Kumar. Meanwhile, all the Indian Institutes of Management (IIMs) have decided to get reports from all the centres which faced problems, analyse them and have a view. Chairman (admissions), IIM-Lucknow, Prof Himanshu Rai said that very soon all the IIMs would provide their views on the problems and how they be tackled. (This article appeared in the Times of India dated 09 December 2009.)

82

83

An EDGE - ValueNotes Report

An initiative by

Supported by

84