Quality
in e-Learning: a Critical Issue
 Assuring
quality is obviously a key concern in many aspects
of learning, education and training
so is it especially crucial in e-Learning? In this
article Adrian Snook sets out a personal view of
the key challenges and highlights the critical
importance of getting it right.
Introduction
Why is Quality Key?
Psychological Impact
Scale of Up-Front Investment
Complex
Development Processes
Observations on Current Constraints
Skill Shortages
The
Trend to Blend
Market
Education
The
Missing Yardstick
Adapting
to New Business Models
New
Realities
Some
Useful Approaches to Assuring Quality
 Introduction
Several
commentators have said over the last year that
e-Learning is just two years into what could be
a 10 year development cycle. Setting aside the
fact that this assertion ignores the huge advances
previously made by the CBT (Computer Based Training)
community prior to 2000, I am in general agreement
with this notion. -One of the key milestones in
the development of any new paradigm is always the
evolution of key definitions for best practice
on the basis of trial and error. In time, where
best practice is found to have critical significance,
this needs to result in the development and widespread
acceptance of formal standards.
Much of the initial groundwork in relation to the
definition of best practice has now been done. In
the late 1990's pioneering organisations like the
UK Institute of IT Training evolved the first generations
of e-Learning standards.
In building on the foundation of these first-generation
standards it is vital to define a way forward that
does not result in reinvention of the wheel and ensures
that those already embracing existing standards are
not alienated.
This article sets out a personal view of some of
the key challenges in when addressing the issue of
Quality Assurance in the context of e-Learning, and
highlights the critical importance of getting it
right first time.
Why
is Quality Key?
Assuring
quality is obviously a key concern in many aspects
of learning, education and training, so why should
it be especially crucial in relation to e-Learning?
Te-Learners, as with other distance learners, are
working in isolation with limited or sometimes
non-existent human support. This means that the
first impact of any failure in the providers
Quality Assurance regime falls directly on the
e-Learner/customer.
By default the e-Learner/customer therefore becomes
directly responsible for:
- Identifying the error
- Determining that the error is generated by the
course-provider and is not a user-error,
- Reporting the error in order for remedial action
to take place
- The time and cost implications of leaving the
learning process to do the above.
When an e-Learner encounters errors caused by a
failure in a providers Quality Assurance regime the
impact might be immediately evident or not become
evident until the learner undertakes an assessed
outcome. Either way there is rarely a safety net
in place to detect the problem.
Psychological
Impact
Partly as a result of isolation e-Learners who fail
to achieve their learning objectives as a result
of poor quality e-learning could conceivably
blame themselves rather than the courseware or
course provider, perhaps to be lost to learning
altogether.
This is far less likely in classroom based learning
where there is teacher and mentor support, supplemented
by the ability to benchmark your progress against
fellow students learning via the same resources.
For many this is a completely new form of learning.
Learners need constant reassurance that e-Learning
can provide a comprehensive and satisfying learning
experience with real benefits over traditional forms
of learning. Once this faith is shaken, it might
never be regained.
Scale
of Up-Front Investment
Up front development costs for e-Learning are typically
much higher than for other forms of learning.
If an effective Quality Assurance regime is not
in place then failure costs can be significant and
financially crippling. The cost associated with re-working
defective e-Learning assets, reintegrating them and
re-testing is often significant. In the case of assets
like video the cost of re-work can equate to the
cost of origination.
Complex
Development Processes
e-Learning
production has now evolved into complex multidisciplinary
activity requiring complex skills in a wide range
of areas, including project management, market
research, pricing, training needs analysis, software
development, media production, learning design
and assessment. There is ample scope for error in all dimensions.
Because
e-Learning is a point of convergence between many
diverse fields an effective
Quality Assurance
regime is urgently required. However this needs
to be holistic and leave no gaps, yet accommodate
the approaches to quality derived from other
areas of specialist practice.
Observations
on Current Constraints
Quality
can be an essentially subjective concept. In the
mainstream world of broadcasting and new media
the term "high quality" is often synonymous
with "high production values".
By contrast the only valid way to assess the true
quality of e-Learning resources is to determine if
the learners making up the target audience have achieved
the learning objectives set for them.
Working from this definition there are a number
of factors that appear to be acting to constrain
the quality of existing e-Learning provision.
Skill
Shortages
There
are identifiable skill shortages in the e-Learning
development industry. The last significant government
investment in developing the skills base of the
CBT Development Industry was a TOPS sponsored pilot
training scheme for computer based training consultants
called Project Author, completed between 1982 and
1984.
The resurgence of interest in technology enabled
modes of learning that occurred in 2000 led in turn
to a growing demand for experienced individuals and
generated significant levels of employment churn
accompanied by rapid salary inflation.
UK and EU Governments are currently placing great
emphasis on e-Learning as a tool to progress their
agenda in areas like the modernisation of the NHS,
the move towards e-Government and workforce development.
The growing shortage of experienced individuals
with relevant e-Learning related knowledge and skill,
underpinned by valid educational qualifications could
well undermine the effectiveness of these policies
or even become their Achilles heel. After all, the
cost of failure in relation to e-Learning initiatives
is high.
Since e-Learning development is fundamentally a
team-based activity, the effectiveness or quality
of an e-Learning programme depends on the weakest
link in the production chain. Superb instructional
design poorly executed will not achieve its goals
for example, nor will poor instructional design perfectly
executed.
The problem is compounded because effective e-Learning
development requires a combination of skills that
rarely co-exist in one individual.
e-Learning exists at a point of convergence between
technology based disciplines and human-centred disciplines.
Individuals gravitate into e-Learning from a wide
range of backgrounds including:
- Software development
- Networking and telecommunications
- Video and audio production
- Teaching or training
- Human resources or personnel
- Web design
- Graphic Design
- Publishing
- Games or simulation
- Management consultancy.
As a result individuals typically possess the knowledge
and skills appropriate to their original role, overlaid
with a veneer of knowledge and skills gathered on
an informal basis from colleagues within an e-Learning
development team.
Effective Quality Assured development requires a
team of well-rounded e-Learning professionals fulfilling
the key roles.
There is a general lack of appreciation of the complexity
and difficulty involved in producing e-learning materials
- there is a mountain to climb, but many people do
not even see the mountain!
The
Trend to Blend
e-Learning
is being increasingly used as part of a blend of
learning provision creating complex interdependencies
between technical elements like text-based workbooks
classrooms, chat-rooms, bulletin boards, synchronous
virtual classrooms, conventional asynchronous content
and services provided by human beings.
It is therefore difficult to isolate the factors
that contribute to quality in e-Learning ; eg high-quality
materials with poor support can lead to less satisfactory
learner outcomes than poorer quality materials with
high level support. The interaction of these and
other aspects is complex.
This makes it increasingly hard to define the boundaries
of a Quality Assurance regime for e-Learning without
addressing broader quality issues.
Market
Education
Many of the observable constraints on Quality are
the result of low levels of market education about
best practice in relation to e-Learning.
There is a generalised lack of understanding amongst
organisations of how to assess their training needs,
the relative appropriateness of e-Learning and the
benefits it can bring.
In larger companies there can be a clash of outlooks
between the ICT professionals who see e-Learning
in terms of systems specifications and HRD professionals
looking at learning content and objectives.
All too often this can result in an emphasis on
what is cosmetically attractive in terms of what
can be done with learning technology, rather than
a focus on achievement of learning objectives - part
of the newness of an emerging paradigm.
The
Missing Yardstick
Traditional
approaches to procurement are predicated on a clear
definition of precisely what is being procured,
supported by appropriate definitions of quality.
This approach works fine when you are buying commodities
like iron ore, flour or potatoes.
When an
organisation commissions the development of a custom-built
e-Learning programme this is sometimes
because they do not have sufficient in-house expertise
to specify or even scope the requirement. The result
is often a hazily worded RFP (Request For Proposal)
that talks in vague terms about requirements for "an
e-Learning programme" or about the delivery
media and technology. Proposals produced by potential
suppliers will therefore invariably offer a bewildering
array of alternative options at different levels
of quoted cost. This often makes it impossible for
the client organisation to effectively compare apples
with apples for the purposes of assessing value for
money. This type of abortive RFP is normally, but
not always scrapped, thereby wasting the time and
resources invested by prospective contractors and
the client organisation alike.
Even when a similar RFP is subsequently issued against
a more detailed specification, many organisations
still have no objective way of measuring the objective
level of quality being offered by competing suppliers.
Correspondingly, suppliers usually have limited mechanisms
for making this information clear to potential clients.
All too often traditional procurement techniques
will be applied to crack the problem. In an attempt
to arrive at a rationally based selection decision
potential suppliers will be asked to set out their
day-rates for various services or for the development
of a learner hour of e-Learning. Alas, there are
no commonly accepted definitions for either of these
terms that are really useful in the context of procurement.
This approach can have many adverse consequences:
- There is a risk that procurement decisions might
tend to favour contractors employing the cheapest
and least experienced untrained staff.
- Unscrupulous contractors are sometimes tempted
to (for example) halve their quoted day rates but
ensure that they take twice the estimated time to
complete the project!
- The deliverables from a completed project can
prove to be effective but surprisingly different
from that anticipated by the client organisation.
- In the absence of a definition for a learner hour
clients can easily be short-changed.
- In the absence of a definition for a learner hour
contractors can potentially over-deliver content
when a more innovative solution might be faster for
learners and equally effective.
In short there is a lack of understanding in buyers
on how to prepare tenders, and lack of skills in
providers in making accurate bids.
Adapting
to New Business Models
The economic models underpinning conventional modes
of learning have evolved into a state of perfection
over 2000 years.
LPublishers have traditionally relied on the services
of independent authors to bring them completed
manuscripts or outlines. In economic terms, the
essential commercial role is that of the Editor,
who informally assesses if a market might exist
for a manuscript or outline. Even if the work
is an outline then the Publisher will often bear
no significant or irreversible cost until the
manuscript is complete.
Having made a decision to publish an author's manuscript,
simple tried-and-tested mechanisms exist to reduce
the Publishers exposure to risk from high publishing
and distribution costs. A short run of books is produced
and shipped. If sales take off then more copies are
printed. If the title fails to sell then the publisher
discounts heavily and in necessary pulps the books
as waste paper. The net risk is limited and easy
to control.
For every best-seller there are always countless
disappointments, but the traditional business model
has developed to accommodate this.
When originating new courses traditional Training
Companies, Schools, Colleges and Universities usually
draw on the hidden intellectual capital of their
trainer/lecturers and their ability to create content
quickly and flexibly.
New courses are sometimes publicised before detailed
course content is even created. If enough students
book places then the course can quickly be produced.
If not, the organisation simply loses their modest
investment in publicity. Course cancellation always
a safe way to avoid exposure to the risk presented
by the relatively high delivery costs associated
with conventional delivery modes.
For a diverse range of reasons, governments over
the last ten years have favoured de-centralisation
and the creation of a multiplicity of fully or partially
privatised educational bodies, agencies and lead
bodies.
Competition between these bodies has been actively
encouraged in the quest for greater efficiency and
at a commercial level competition is growing.
The net result is generally that all of these organisations
have grown increasingly less prepared to co-operate
with other organisations sharing similar/competing
objectives. If resources grow scarcer, this effect
is likely to be magnified proportionally.
There are parallels with the pre-industrial world
of cottage industries, where all aspects of a development
process are undertaken in-house by a multi-skilled
craftsperson. The traditional processes underpinning
traditional Academic institutions, Print-based Publishers
and conventional Training Companies are broadly built
on three common assumptions:
- That the true costs associated with the design
and development of content are either low, not formally
recorded or not incurred directly by the organisation,
- That the elapsed time taken to develop content
is relatively short,
- That the costs associated with implementation,
publishing and delivery are high, relative to the
low development costs previously incurred.
Consequently, the types of organisation described
often exhibit similar characteristics:
- Few processes for managing the development of
complex learning resources and the associated investment,
- Relaxed approaches to market research, because
a small audience is normally sufficient to recover
their limited development costs and because any exposure
to risk is simple to quantify/easy to control.
New
Realities
Unfortunately,
when traditional Academic institutions, Print-based
Publishers and conventional Training Companies
move into the world of on-line learning the established
models that these organisations operate promptly
reverse:
- The true costs associated with the design and
development of content are high, non-recoverable,
formally recorded and typically incurred directly
by the organisation
- The elapsed time taken to develop content is relatively
long
- The costs associated with implementation, publishing
and delivery are low, relative to the high development
costs.
Due to the higher development costs associated with
new media, far larger audiences are required to recoup
the development costs on a per capita basis than
would be the case for conventional modes of learning.
It is not uncommon for organisations moving into
e-Learning to overlook this point or at least completely
underestimate its impact.
Many organisations across the EU are developing courseware
with common learning objectives but for small independent
audiences that fall far short of the optimum economic
break-even point. They are effectively stuck in the
era of the producer-distributor cottage industry. Where a sub-optimal size of target audience is addressed
via e-Learning two potential outcomes occur.
Either:
- A course is developed to acceptable standards
and the provider makes a loss
- A sub-optimal or poor quality course is developed
within the constraints of the budget.
Unless audiences for typical courses exceed a minimum
threshold, e-Learning could remain locked in a world
of mediocrity imposed by limited budgets.
Organisations
moving into the world of e-Learning are often woefully
under prepared for the level of
external co-operation, structural re-organisation
and re-engineering of processes required to assure
quality. e-Learning favours large markets and that
means making some difficult decisions.
Some
Useful Approaches to Assuring Quality
In the late 1990's, the UK Institute of IT Training
(IITT) conducted research into how the corporate
sector was embracing, or was planning to embrace,
e-Learning. The conclusion, broadly in line with
other contemporaneous studies, was that, whilst
e-Learning was recognised as potentially able to
deliver effective learning as part of an overall
training mix, serious barriers to widespread take-up
existed.
The IITT conducted a series of initiatives to address
the issues.
The IITT Code of Practice
The IITT Code of Practice for e-Learning Providers
is the basis for the Institute of Information Technology
Training's E-learning Provider Accreditation Programme
through which the Institute maintains and monitors
a register of approved providers of e-learning.
Organisations that voluntarily undertake to abide
by the terms of this code of practice and can demonstrate
that they do so are eligible to be accredited by
the Institute.
The Code of Practice can be reviewed on-line at:
www.iitt.org.uk/public/standards/e-learningcode.asp
The e-Learners Charter
Directly resulting its 'supply side' standards,
the Institute of IT Training also promotes an e-learner's
charter, which defines what the 'customer', the e-learner,
is entitled to expect from any e-learning experience.
As an e-Learner, you are entitled to expect your
course to:
- Clearly state what it is designed to achieve and
how it will be of benefit to you
- Make clear what is required of you if you are
to complete it successfully
- Allow you to control what you do, where, when
and for how long
- Be engaging and stimulating
- Recognise and respond to your individual characteristics
and preferences
- Help you to relate new learning to your past experience
and the issues and problems with which you are currently
faced
- Recognise that you're more likely to learn by
doing things than by being presented to
- Work smoothly and reliably on the hardware and
software platforms for which it is designed
- Neither exclude nor discriminate against you because
of disability, race, gender or age
- Be easy to use
- Allow you to monitor and assess your level of
progress
- Stimulate and facilitate collaboration with fellow
learners
- Provide you with human support to recognise your
successes and help you with any difficulties
- Provide this support in an accessible and responsive
manner.
The e-Learners Charter can be reviewed on-line at:
www.iitt.org.uk/public/standards/e-learncharter.asp
IITT Standards for e-Learning Materials
For e-learning to fulfil its promise, learning materials
have to be designed so as to facilitate learning
within a stimulating whilst enjoyable environment.
The IITT Standards for Learning Materials
focus on seven key areas: integral learner support;
content;
learning design; pre & post assessment; navigation;
usability; media quality and technical quality. The
e-learning materials standards are applied when assessing
an applicant for accreditation under the Institute's
e-learning provider programme.
The IITT Standards for Learning Materials can be
viewed on-line at:
www.iitt.org.uk/public/standards/e-learningmatsstand.asp
The Certified e-Learning Professional
Since e-Learning development is fundamentally a
team-based activity, the effectiveness or quality
of an e-Learning programme depends on the weakest
link in the production chain.
The problem is compounded because effective e-Learning
implementation requires a combination of skills that
rarely co-exist in one individual.
The aim of the Certified e-Learning Professional
Programme is to provide organisations with well-rounded
e-Learning professionals capable of applying the
necessary knowledge and skill to plan, develop, implement
and support e-Learning effectively.
The first step for the CeLP initiative was to structure
the various domains of knowledge and skill into a
set of competency frameworks for the key roles in
the e-Learning development team. An e-Learning Standards
Panel was formed in September 1999, chaired by Clive
Shepherd, a Director of both the Institute and The
Training Foundation and a highly respected industry
figure.
The panel was made up of representatives from various
sector interests and published its Standards for
e-Learning in March 2000. This included a Competency
Frameworks for developers and tutors, followed in
due course by Frameworks for e-trainers, managers
and consultants.
The competency frameworks devised by the panel formed
the detailed learning objective pinning the Certified
e-Learning Professional (CeLP) courses and the assessment
criteria used to assess competence.
From these inputs, the IITT's Certification Committee
produced a Curriculum Specification, designed to
deliver the defined competencies to each of the professional
tracks.
You can find out more about the Certified e-Learning
Professional qualifications from the programme Portal:
www.elearningprofessional.com
This
article is based on a Paper presented to the e-Learn
Accredit Conference 2002
|
