Examining Anti-Plagiarism
Software:
Choosing the Right Tool
The CMC Anti-Plagiarism Software Survey
Cynthia Humes, Associate Dean of Faculty for Academic Computing
Jason Stiffler, Interim Director of the Writing Center
Micheal Malsed, Assistant Director of Student Computing
Abstract:
Parallel to the collection of survey data regarding the software,
instructors in the Literature Department have been testing a combination of Turnitin.com's detection services, and Prentice Hall's
educational software. Results of this program are still preliminary, but
indicate a pattern of support for the findings of the survey results. After one
semester of employment, the program does appear to be helping faculty to detect
student plagiarism, and students seem comfortable both with the detection
program, and with efforts to better teach them the rules of academic honesty.
Introduction
For the past year,
To begin this process of exploration, a committee was assigned in late 2002
consisting of several students, the Associate Dean for Academic Computing, the
Chair of the Literature Department, the Assistant Director of Student
Computing, and an Educational Technology Specialist. This committee examined a
wide range of software packages in an informal process. The committee concluded
that an effective anti-plagiarism program should include both an
educational/preventative element (to help students better understand the nature
and consequences of academic dishonesty), and a detection element (to assist
faculty in identifying acts of plagiarism when they occur). The committee
recommended that a pilot implementation of educational software from Prentice Hall, and detection software from Turnitin.com begin
immediately. It further recommended that a more formal process of software
review be undertaken, to more accurately assess student and faculty opinions of
various packages relevant to the problem of academic dishonesty. That software
review process has been undertaken during 2003 by a team which grew out of the
original committee, composed of the Associate Dean, the Chair of Literature,
the Director of the Writing Center, the Assistant Director of Student
Computing, a faculty representative from the Psychology Department specializing
in learning and cognition, an Educational Technology Specialist, and a Staff
Tutor from the Writing Center.
This report presents the results of the efforts of this committee to provide
a quantitative assessment of faculty and student satisfaction with a broad
range of anti-plagiarism software. It also provides a discussion of the first
semester of CMC's adoption of turnitin.com, and the Prentice Hall plagiarism
teaching module. The report is divided into individual sections concerning
plagiarism education tools, plagiarism detection tools, and the current pilot
program at CMC.
Technology-Based Approaches to Plagiarism Instruction.
The committee surveyed available options for training students about
plagiarism and academic honesty, and compiled a list of third-party tools and
academic websites that appeared to be providing leadership in the field. This
list included the following:
GPTeach, by Glatt
Plagiarism Services
MITT: The Multimedia Integrity Teaching Tool
The Prentice Hall Companion Website "Understanding Plagiarism"
The "Avoiding Plagiarism" page at Purdue's Online Writing Lab
The "Plagiarism" page at Indiana University
The University of Michigan's anti-plagiarism site
Plagiarism.org - the education arm of Turnitin.com
A brief discussion of each of these tools is available in Appendix VI.
Six of the tools on this list (all but MITT) were placed before a team of
roughly 60 student test subjects, who were asked to use the tools to teach
themselves about plagiarism and then to complete a survey indicating their
confidence in the instruction they were receiving. Not all students completed
survey for all software packages - in general, there were roughly 20 responses
per program. The committee had originally intended to include MITT in the
student survey - indeed, MITT had been the focus of more testing than any other
package (with unsatisfactory results) during the informal testing process
employed prior to setting up the pilot study in 2002. During that early
testing, the committee was impressed by the scope of MITT's
training, but found the interface to be problematic. However, the committee was
unable to obtain an evaluation copy of the most recent version of the software
in time for inclusion in this study, and we felt that it would be unfair to the
program's developers at Ball State University - and potentially misleading to
our report - were we to judge their teaching tool based upon an outdated
version of the software, which we expect has been updated significantly since
we reviewed it a year ago.
The survey students were selected from among student employees of various
divisions of the Educational Technology Services department. They range across
all four classes, and represent a range of skill-levels in both technology and
writing, as included among the survey sample were a significant number of new
student hires, who have not yet been trained in technology, and who in most
cases are currently enrolled in their first college writing course. There is a
slight male-leaning gender bias in the sample, as well as a bias in favor of
underclassmen. This second bias likely works in favor of the survey: the target
audience of these instructional tools is mostly composed of freshmen and transfer
sophomores.
Table 1: Overall Survey Results
The students who took part in the survey process were asked to indicate their
level of satisfaction on a 5-point Likert scale
(1=strong dissatisfaction, 5=strong satisfaction) in the following areas:
1) Did the interface seem intuitive?
2) How well did the software teach you about ethical issues in academic
honesty?
3) Did the software provide you with useful information about plagiarism?
4) Did the software help you to understand the harm done to the college
community by plagiarism?
5) Did the software do a good job of teaching you about other areas of academic
honesty besides plagiarism?
6) Would this program be useful to you, as a student?
7) Was the time commitment posed by this program reasonable?
(A listing of the exact questions posed on the survey is available in
Appendix V). Students were also asked an eighth question concerning the
usability of the documentation of the programs. However, most students found
all of the programs easy enough to use that they made little reference to the
documentation - as a result, only a few students were able to provide feedback
concerning the usability of documentation. For this reason, the documentation
question was removed from the overall study. A table itemizing the responses we
received to that eighth question is available in the appendices.
Trimmed means were taken for each of the seven questions on the survey, the
individual results of which are available in Appendix I. To provide a measure
of the overall student software preference, an evenly-weighted average was
taken of the mean score received on each of the seven questions by each
software package. This overall mean score is presented above, in Table 1.
Overall, the "Understanding Plagiarism" package from Prentice Hall
scored best, earning an overall mean rating of 3.79. By contrast, the GPTeach package from Glatt earned
the lowest marks, with a mean score of 3.06. Overall, this indicates that all
six programs are seen in reasonably similar terms by the students; the overall
score for all six being within one point. While the students did not report any
statistically significant preference among the mid-range scoring packages
(Plagiarism.org, and the websites at
Table 2: Overall student preference, organized by type
Discussions with students involved with the testing indicated (anecdotally)
that they perceived a qualitative difference between those learning tools which
simply presented them with information (the university websites, and
Turnitin.com), and those which actively quizzed them to reinforce knowledge (GPTeach and Prentice Hall). While this
difference would likely be invisible in the classroom (as faculty develop and
implement their own assessment programs with which to measure student
comprehension), it does point to a possible selection effect in the testing
process. Thus, Table 2 presents the same data, color-coded to show
overall student preferences for programs with a testing component (purple base)
and programs without one (blue base). Following the pattern in Table 1,
high-scoring tools in each field are capped with a green gradient, and
low-scoring tools in each field are capped with a red gradient. While Purdue's
website scores slightly higher than those at
Conclusions
Based upon the results of this survey, two primary conclusions are apparent.
First, given that students judged no teaching tool as more than marginally
acceptable as a means of learning about plagiarism, it seems clear that there
is room for significant development in online and technology-based academic
honesty teaching tools. While it is possible that the latest version of MITT
(which was unavailable for inclusion in the survey) may help to fill this need,
the general opinion of the review committee is that the plagiarism training
field as a whole would benefit from new offerings, or from improved versions of
some of the existing software.
Second, the results do indicate that, while no product is perfect in the field,
there are measurable differences in student satisfaction with the six programs
tested. These results indicate that the best available program at present is
that offered by Prentice Hall, which tests as either measurably preferable, or
tending toward preferable, as compared with all competitors. Similarly, the
survey seems to agitate against the adoption of Glatt's
GPTeach program. Particularly given that Prentice
Hall offers its learning module for free, while Glatt
charges a significant licensing fee for GPTeach, the Glatt package seems a poor choice, even given the limited
success of the field. Finally, if a faculty member is looking for a website to
provide reference materials for students, the only statistically measurable
advice this survey offers is that the materials at
Electronic Plagiarism Detection
The committee reviewed major plagiarism detection services, and compiled a
listing of major automated packages and methodologies for performing wide-net
online searches for plagiarized material. The committee identified the
following services and programs:
The Essay Verification Engine, v2 (Eve2) by Canexus
Turnitin.com, by iParadigms
The University of Virginia's WCopyFind, in
conjunction with Google
YAP, from the University of Sydney
MOSS, from UC Berkeley
SCAM, from Stanford University
Glatt Plagiarism Screening Program
IntegriGuard
JCIS Plagiarism Detection Service
However, many of these programs and services were immediately eliminated as
unsuitable to the full college environment. YAP and MOSS each focus exclusively
on plagiarism of software code, and were thus eliminated from consideration.
SCAM, from Stanford, appears to have been a student project - and is no longer
available or supported. The freely-testable version of Glatt's
software indicates on its website that it is only suitable for advisory
purposes, and will not provide truly accurate reports. IntegriGuard
has gone out of business. The JCIS Plagiarism Detection Service is only
available to institutions in
Elimination of these services left the committee with three methods to
choose from: Eve2, Turnitin.com, and web searches (we used Google) enhanced by
the report-generating capabilities of WCopyFind.
Details of each of these detection methodologies may be found in Appendix VII.
The faculty were asked to complete a 17-question survey rating their
satisfaction with the software on a 5-point Likert
scale (1=strong dissatisfaction, 5=strong satisfaction). The specific questions
from this survey may be found in Appendix V. Additionally, Appendix III
provides graphical breakdowns of responses to each of the 17 questions, for
reference. Overall, the team expressed a strong, consistent, and statistically
significant preference for Turnitin.com, followed by WCopyfind,
and then Eve2. This pattern emerges not only in the overall score for the
programs, but in 16 of the 17 individual questions. The only question for which
this pattern did not hold concerned the level of technical ability (seen as a
negative) which the software required of the user. In this one area, Eve2 slighly out-scored WCopyFind on
average, though Turnitin.com still led the field in this area. Statistical
measures of confidence in the difference between means for these tools exceeded
99.9% in all comparisons in the overall study. Statistical notes on the results
from individual questions may be found in Appendix IV.
Table 3: Overall Faculty/Staff Preference in Detection Software
In addition to the 17 quantitative survey questions, each participant was also
asked to provide anecdotal responses to two questions: "How was the
"look and feel" of the software?" and "Please provide any
comments". The comments provided in response to these two questions were
surprisingly consistent, and provided useful insight into the reasoning of the
team in its preference for Turnitin.com. The team indicated that Eve2 suffered
in comparison to other products in that it generated a less-usable plagiarism
report, and sometimes seemed to generate false-positives in that report which
were then more difficult to track down because the report was hard to use. By
comparison, the team indicated that Turnitin.com provided a very usable report,
and one containing results which appeared to be highly reliable. In the
mid-range, team members reported satisfaction with the reports generated by WCopyFind, but rated the University of Virginia method
lower than Turnitin.com overall, as it did not include an automated searching
tool. Generally the team members were comfortable using Google to seek out
plagiarism on the internet, but found it to be a needlessly time-consuming
process compared to the ease of Turnitin's automated
search.
A final interesting note concerns the readiness of faculty to adopt these
software packages in their courses, or to recommend them to others. The data on
these two points are posted in tables 4 and 5, below. Overall, the team members
would not be comfortable recommending any tool except Turnitin.com to their
peers, nor would they be comfortable adopting WCopyFind
themselves. By comparison, the team indicated that they would be extremely
comfortable both adopting and recommending Turnitin.com, offering unanimous
strong support for that software. This combination seems to suggest that
turnitin.com is the only tool among those tested that would be likely to
succeed in earning adoption at the College without a significant incentive
program.
Table 4: Would Recommend To Others
Table 5: Would Adopt in Your Own Class
Conclusions
The results of the survey indicate both that faculty and support staff at
CMC are highly satisfied with Turnitin.com as a method of detecting online
plagiarism - and that they strongly prefer it to the other two major means of
detecting plagiarism online. In the anecdotal section, several of the
respondents did indicate that they felt the service would be of more use to
faculty users if it were offered in conjunction with short workshops of other
assistance to help users "get started" with the software. In general,
the opinion of the team members was that it was quite simple to use, once you learn
the basic interface.
Similarly, the team indicated quite strongly that Eve2 is not a sufficient
tool for the CMC community. The poor report structure, coupled with the need to
make close use of that report to check the consistency of the search results
the software generates, serves as a strong barrier to success, particularly for
users less familiar with the intricacies of plagiarism searches.
CMC's Pilot Program
Since December 2002, CMC has been conducting a pilot implementation of
Prentice Hall's teaching software and Turnitin.com's
detection software, to test the feasibility of implementing an academic honesty
module as a component of CMC's "Fluency in Information Technology"
(FITness) program, which attempts to incorporate technology skills across the
curriculum at the College. The FITness program has listed instruction in
Digital Ethics as a key component of the technology skills curriculum, and the
effort to fight online plagiarism seems to work well with this goal.
During Spring Semester 2003, the Prentice Hall software was employed in
three sections of Lit 10 - Composition and Literary Analysis. All three
instructors using this tool indicated satisfaction with it as a source of
information on plagiarism. The instructors were particularly happy with the
fact that the Prentice Hall website sends email copies of the comprehension
tests to the course instructor, allowing him or her to track student
understanding of academic honesty issues. For the purposes of this study,
supplemental quizzes were constructed using the College's courseware package
(WebCT), thus avoiding the problems posed by several sections in the Prentice
Hall module for which no self test is included. Each of the three instructors
also made use of Turnitin.com for spot checks to detect plagiarism. One
plagiarist was caught among the three courses.
Students in one course section were subsequently polled to determine their
satisfaction level with the instruction offered by the software. Students were
asked their opinion of the utility of the Prentice Hall system as a learning
tool, as measured in three areas (quality of the module, effectiveness of the self-tests, and helpfulness of the reference materials).
Using a 5-point Likert scale (1=strong
dissatisfaction; 5=strong satisfaction). Overall, the students were ambivalent
about the quality of the learning software was a whole, and felt only
marginally positive about the self-tests, and only
marginally negative about the reference materials (see Table 6).
Table 6: Student Perception of Prentice Hall in pilot testing
It is interesting to note that if these results are tested for significance of
difference between this result and an expected outcome of perfect neutrality (a
score of 3), we can only say with marginal 80% confidence that there is any
statistically significant difference at all. Thus, the students cannot be said
to have any strong opinion either way regarding their attitude toward the
software.
Students were also asked to indicate their comfort level with the amount of
time required to complete the learning module. Students indicated their
sentiment on a 5-point Likert scale, where 1=too
little time, 3=right amount of time, and 5=too much time was required to
complete the module. The overall mean score of 3.53 indicates that the module
may be slightly longer than some students would prefer, but is basically of a
reasonable length. These results are presented in the histogram, below.
Table 7: Student Perception of assignment length for Prentice Hall in pilot
testing
Running a t-test to compare the mean student response of 3.53 against an
expected score of "just right (a score of 3), we can say with 98%
confidence that the students did find the assignment to be slightly longer than
they might have preferred.
Finally, students were asked to comment on their comfort level with the
professors running spot checks through Turnitin.com on papers submitted in the
class. Only qualitative information was collected on this point. Table 8
provides a histogram denoting the frequency of various responses (the full
question text - along with the full text of all questions asked - is available
in Appendix IX). Students were provided with six "comments" to choose
from, and were allowed to select as many options as they felt applied to their
opinion.
Table 8: Student comments on Turnitin.com spot checks during pilot study
Obviously any information we can draw from this table will be purely
anecdotal, though it is interesting to note that as many people felt that their
peers needed policing, as felt that their privacy had been invaded. There was
also a nearly even split on whether the students at the College are
sufficiently dishonest to justify plagiarism checking. It is encouraging that at
least one student was discouraged from plagiarizing as a result of the efforts
in the pilot study.
Conclusions
The results of the pilot study are useful for comparison to the results of
the more recent survey of student, faculty and staff opinions regarding
anti-plagiarism software. Perhaps most interestingly, whereas students felt
ambivalent about the usefulness of the Prentice Hall software in the pilot
study, they reported a statistically measurable (if still lukewarm) positive
reaction to the software when it was placed in comparison with competitor
products - with a greater than 99.9% measure of confidence. This indicates that
the ambivalent reaction of the students to the Prentice Hall software in the
pilot study might have been significantly worse, had a competitor product such
as Glatt been used in its place.
The insights this pilot study provides into student perceptions of
plagiarism detection software are also revealing. The significant number of
students indicating that they felt privacy concerns with the spot checks
indicates that further work exploring these concerns might be useful.
Specifically, it would be interesting to test student reactions to Turnitin's policy of archiving past submissions for future
anti-plagiarism checks. This policy has raised some alarms in the academic
community regarding intellectual property rights, and might well turn out to be
a source of student concern, as well.
In summary, the pilot study seems to confirm that students do not
particularly dislike the Prentice Hall software, and are as likely to
appreciate spot checks through Turnitin.com as they are to resent them. This
indicates that campus-wide implementation of these two software packages is
feasible, should the faculty determine that such a program is in keeping with
the mission of the College. The results of the survey of academic honesty
teaching software do seem to indicate that further development in the field
might be appropriate before any such adoption is likely to occur.
Overall Recommendations
Overall, the results of this study suggest that, while there is room for
improvement in the academic honesty teaching field, post secondary institutions
interested in adopting such software may wish to consider the free
"Understanding Plagiarism" module from Prentice Hall above competing
products in the field. While it tests only marginally better than competing
free resources available online thorugh Purdue and iParadigms, student reaction during the pilot study in the
CMC Literature Department suggests that the slight preference students grant it
in comparison to other products may be important in preventing a negative
student reaction to the program as it is implemented. Since the pilot program
suggests that positive response to the software declines in the absence of
comparison, it seems reasonable to suggest that any margin or preferability is worth seeking. In any case, the report
strongly suggests that free teaching modules like the Prentice Hall package are
preferable to lower-ranking, fee-based services like Glatt's
GPTeach.
In the area of plagiairism detection, Turnitin.com
is a clear winner, as the only test subject throughout the survey to elicit a
strongly positive response from the survey takers. In implementing Turnitin's services, anecdotal evidence suggests that some
form of formal support mechanism may be helpful, and that privacy concerns of
the student body should be addressed.
Appendices published under separate cover