University of Oxford Logo University of OxfordSoftware Engineering - Home
On Facebook
Facebook
Follow us on twitter
Twitter
Linked in
Linked in
Google plus
Google plus
Digg
Digg
Pinterest
Pinterest
Stumble Upon
Stumble Upon

Examination Conventions

These Conventions were approved and adopted at the meeting of the Supervisory Committee in Trinity term 2017. No student will be disadvantaged by changes to Conventions which occur during their course of study or period of registration.

Assessment Processes

The University appoints a board of seven examiners to oversee all assessment activities on the Programme: two of these examiners are external to the University; another is the Programme Director; the remaining four—including the chair—are drawn from the academic staff of the Software Engineering Programme, and their appointment is subject to the review and approval of the University Proctors. For the academic year 2016/2017, the examiners are:

External Examiners:

Dr Steve King
Senior Lecturer,
Department of Computer Science,
University of York.

Dr Tom Chothia
Senior Lecturer in Computer Security,
School of Computer Science,
University of Birmingham.

Internal Examiners: Dr Ivan Martinovic (chair); Prof. Jim Davies (Programme Director); Dr Andrew Simpson; Prof. Cas Cremers; Dr Andrew Markham.

The Board meets three times a year—in each of the three University terms—to review the assessment of assignments submitted since the last meeting, consider the performance of candidates who have entered the final examination for an award.

In reaching their decision, they will follow the conventions described below.

Written Assignments

Examination Process

Each course of the Programme is examined by a written assignment. The assignment is both summative and formative—as well as measuring the student's grasp of the material, the very exercise of completing the assignment is an important part of the learning process—so the reports are taken very seriously by assessors, and greatly appreciated by students. The examination follows a comprehensive examination process with six stages:

  1. preparation: the examination paper is prepared, together with a set of assessment criteria;
  2. review: the examination paper and assessment criteria are considered by the examiners, who may request modifications;
  3. assessment: the students' submissions are assessed by two markers, who propose grades, and produce reports;
  4. moderation: the submissions, grades, and reports are considered by the moderators/examiners, who oversee the reconciliation of different views, and may request re-assessment, or adjustment of grades;
  5. pre-release: the moderation is reviewed and preliminary grades and reports are released;
  6. confirmation: the moderated grades and reports are considered and confirmed by the examiners at the next Examiners' meeting.

Each assignment has clearly stated objectives, accompanied by a list of assessment criteria, an indication of expected extent (typically the equivalent of 20 pages of text), and a statement of maximal length. At the review stage, an examiner will check that the objectives, criteria, and extent are appropriate: in particular, they will check that the assignment could be completed by a competent student within 25 hours.

Submissions are marked by two appointed assessors: normally the lead lecturer, and the teaching assistant in the subject concerned. Each assessor proposes a mark and a report for each submission. The two markers' reports are then moderated by an examiner, who will take steps to resolve any differences.

  • Where the difference in grades is less than 10 marks, and does not cross any fail/pass/distinction boundary, then the examiner will invite the markers to agree upon an intermediate grade.
  • If the difference in grades is greater than 10 marks, or crosses a fail/pass/distinction boundary, the markers are invited to resolve the conflict by discussion—after viewing each other's reports. If the markers are able to agree on a reconciled mark, this is recorded. If appropriate, the examiner will request updated versions of the reports to reflect their findings. If the markers cannot agree, further reports are commissioned: usually the external examiner is asked to review such cases prior to the Examiners' meeting.
  • All the information—including original and revised reports—is made available to the external examiner. At the Examiners' meeting the results of the moderation are reviewed and approved, producing a final mark and report to the student.

It is not possible to re-take an assignment. Instead, students may register for the assignment part of a later module in the same subject, and any submission that they make will be considered in the normal way. Up to three such subsequent resubmissions are permitted for the MSc, two for the PGDip, and one for the PGCert.

Marking Guidelines

All submissions are awarded a numerical grade between 0 and 100, interpreted as follows:

  • a mark between 0 and 49 denotes a fail: a student will have: demonstrated only a limited ability to comprehend the subject, and little or no ability to apply the concepts, tools, and techniques; failed to attempt, or to complete successfully, certain essential aspects of the assignment task.
  • a mark between 50 and 69 denotes a pass: a student will have: demonstrated understanding of the subject, and an ability to apply those concepts, tools, and techniques relevant to the assignment task; shown that they are able to complete the essential aspects of the assignment task successfully; (which aspects, or combinations of aspects, are deemed essential to a demonstration of competence and understanding will depend upon the subject area and the nature of the task).
  • a mark between 70 and 100 denotes excellence: a student will have: demonstrated a convincing understanding of the subject, encompassing at least that part of the subject addressed in the assignment; shown that they are able to complete the whole of the assignment task successfully, or that they would be able to do so with relative ease following a small amount of consultation or feedback; provided some evidence that they are capable of reflecting upon—understanding the reasons for, and the context of—the assignment task itself.

Typographical errors, minor misunderstandings regarding syntax and semantics, small omissions, or confusion regarding the precise expectations of the examiners, need not detract from an assessment of excellence.

The assessors are asked to consider the following guidelines, in terms of the application of core concepts, ideas, techniques, methods, or tools, the quality of explanation or reflection, and the correspondence to a model solution.

  • 0-9: indicates the lack of any serious attempt at the assignment.
  • 10-19: indicates a genuine attempt at the assignment, but one that suggests that the student was not ready to attend a course in this subject, or that they were unable to give the course or assignment sufficient attention: not only might the student be advised to re-take the course, but they might also wish to consider their study plans overall.
  • 20-29: indicates a lack of appreciation and understanding, to the extent that the student would be advised to attend the course again before attempting any subsequent assignment in the same subject, or registering for any course that assumed prior knowledge and competence in this area.
  • 30-39: indicates a clear misunderstanding of some essential aspect of the subject, or that a significant part of the assignment has been omitted or poorly attempted; other aspects, and other parts, however, demonstrate a good appreciation of concepts and techniques.
  • 40-49: indicates a good appreciation of the concepts taught in the course, and an ability to apply the techniques presented, but not to the extent that would safely represent mastery of the subject; there is no 'dangerous nonsense' here, but neither is there sufficient evidence of ability and understanding.
  • 50-59: indicates knowledge and understanding of the concepts, as demonstrated through routine application to a situation that, though novel, fits within a standard prescription; with adequate documentation, a factual or minimal explanation of work presented; satisfying the basic criteria for competence.
  • 60-69: indicates an appreciation of the applicability or effectiveness of techniques, as demonstrated by the selection of appropriate techniques in novel situations; with a good explanation of the approach taken, and the motivation for choices made; addressing most if not all of the assessment criteria.
  • 70-79: indicates knowledge and understanding of the underlying mechanisms determining the applicability and effectiveness of different techniques, as demonstrated by deeper, reasoned arguments in support of selection; an account of the context of the work and clear consideration of alternative approaches; close, but not good enough to serve as a model solution.
  • 80-89: indicates a wider appreciation of applicability and effectiveness, as demonstrated by an understanding of the alternative perspectives, and of the scope or possibilities for improvement, integration, or innovation; good enough to serve as a model solution, such as might be distributed as an indication of how the assignment task ought to be addressed.
  • 90-100: indicates an ability to successfully extend or innovate, as demonstrated by an outline of a new argument, a new approach, a new solution, or a suggested extension to theory or practice; a possible improvement upon the model solution as originally envisaged by the assessor, either in terms of presentation, detail, or clarity, or through the inclusion of some aspect not previously considered.

Rules for Late Submission

With effect from September 2016, the rules for assignment submission are as follows. These rules will be incorporated into the Examination Conventions at the first opportunity.

  • A student can revise and re-upload their assignment submission at any time before the deadline.

  • Late submissions will be accepted, up to 144 hours after the deadline. Late submissions are generally penalized, according to the following tariff:

    Penalty
    up to 4 hours 1 mark
    up to 24 hours 5 marks
    upto 48 hours 10 marks
    up to 72 hours 20 marks
    up to 96 hours 30 marks
    up to 120 hours 40 marks
    up to 144 hours 50 marks

  • If the penalty exceeds the earned mark, the resulting grade will be zero rather than negative.

  • It is the timestamp on the submission system that determines lateness.

  • If a student submits late, the Proctors will write to them via their Oxford email address to alert them to the consequences, and to ask whether there are extenuating circumstances. The student has five working days in which to reply. If the Proctors are satisfied, they may then instruct the Examiners to waive any late penalty.

  • A student may withdraw from any assignment, without penalty, as long as they do so before the submission deadline. They will then be able to sign up for a later assignment. If a student does not formally withdraw, and then does not submit, this will count as a failure to meet the requirements for the module; the Programme cannot then grant another attempt at the assignment.

Dissertations

Assessment Process

To be awarded an MSc, a student also needs to write a dissertation based upon a project of their own design. Dissertations will be assessed once a candidate has completed the necessary assignment submissions for the MSc award. The assessment of dissertations follows a well-defined process with four stages:

  1. assessment: each dissertation is assessed by two markers, who propose grades, and produce reports;
  2. reconciliation: in case of conflicting perspectives, an attempt is made to reconcile the two views;
  3. aggregation: the grades and reports are discussed and finalised at the Examiners' meeting;
  4. release: the finalised grades and reports are released.

Each dissertation is marked independently by an appointed assessor and an internal examiner, who each supply a written report and a proposed mark; the assessor and the examiner must be different individuals, and they must be different from the supervisor of the dissertation.

The supervisor writes an independent report commenting on matters that may not be evident from the dissertation, such as difficulties in accessing materials and degree of engagement with the supervisor.

All three reports are then considered by the Chair of Examiners, who will take steps to resolve any differences. The Chair will invite the two markers to read all three reports, discuss and resolve any differences of opinion, and agree on a reconciled grade.

If the markers are able to agree on a reconciled mark, this is recorded. If appropriate, the Chair will request updated versions of the reports to reflect their findings. If the markers cannot agree, further reports are commissioned: usually the external examiner is asked to review such a dissertation prior to the Examiners' meeting. To ensure consistency and fairness, the external examiner is also asked to consider other dissertations within the same band.

All the information — including original and revised reports — is made available to the external examiner. At the Examiners' meeting both contentious and uncontentious cases are discussed and resolved, producing a final mark and report to the student.

Marking Guidelines

A slightly different interpretation of the numerical grades is applied in the assessment of the project and dissertation, where students set and report upon their own assignment tasks:

  • a mark between 0 and 49 denotes a fail: a student will have demonstrated only a limited ability to comprehend the chosen domain, and little or no ability to select or apply appropriate principles and techniques to the chosen task.
  • a mark between 50 and 69 denotes a pass: a student will have demonstrated an ability to select and apply appropriate principles and techniques, and to reflect upon their application in a specific domain.
  • a mark between 70 and 100 denotes excellence: a student will have demonstrated an ability: to select, apply, extend, and adapt these principles and techniques to novel situations; to evaluate the potential for application across different domains.

The application should be sufficiently challenging as to allow scope for a convincing demonstration of professional competence and academic ability at Masters' level, and the student's understanding must be clearly demonstrated through a coherent, logical presentation of context, application, and reflection.

The assessors are asked to consider a set of guidelines, in terms of the application of concepts, ideas, techniques, methods, or tools, and the quality of explanation or reflection. More specific guidelines are given for each 10-mark range:

  • 0-9: indicates the lack of any serious attempt at the dissertation.
  • 10-19: indicates a genuine attempt, but one where the student has failed to provide minor evidence of any understanding or relevant material pertaining to the subject.
  • 20-29: indicates a lack of appreciation and understanding, the student has been unable to demonstrate an ability to apply concepts, tools or techniques in almost all areas of the topic.
  • 30-39: indicates a clear misunderstanding of some essential aspect of the subject, or that the student has demonstrated only a limited understanding of the subject, failing to demonstrate competence in essential aspects of the topic.
  • 40-49: indicates a good appreciation of the concepts or techniques but not to the extent that would safely represent mastery of the subject; there is no 'dangerous nonsense' here, but neither is there sufficient evidence of ability and understanding.
  • 50-59: indicates knowledge and understanding of the concepts, as demonstrated through routine application to a situation that, though novel, fits within a standard prescription; with adequate documentation, a factual or minimal explanation of work presented; satisfying the basic criteria for competence.
  • 60-69: indicates an appreciation of the applicability or effectiveness of techniques, as demonstrated by the selection of appropriate techniques in original situations; with a good explanation of the approach taken, and the motivation for choices made.
  • 70-79: indicates knowledge and understanding of the underlying mechanisms determining the applicability and effectiveness of different techniques, as demonstrated by deeper, reasoned arguments in support of selection; an account of the context of the work and clear consideration of alternative approaches; close, but not good enough to serve as a model for how this should be done.
  • 80-89: indicates a wider appreciation of applicability and effectiveness, as demonstrated by an understanding of the alternative perspectives, and of the scope or possibilities for improvement, integration, or innovation; good enough to serve as a model or exemplar.
  • 90-100: indicates an ability to successfully extend or innovate the underlying principles or techniques, independently of specific application domain; dissertations marked at this level are exceeding expectations, in terms of depth and clarity, or through the presentation of some significant insight or contribution.

Awards

As students are able to choose the point at which they enter the examination, it is important that they are able to determine whether or not they have done enough, in principle, to satisfy the examiners. The examination conventions for the Programme present sets of sufficient conditions for passing or for achieving a distinction for excellence: while the Examiners have the discretion to set the conventions aside in individual cases, this gives a clear indication to both student and supervisor of the likely outcome.

To pass the MSc, it is normally sufficient to:

  • attend courses in 10 different subjects;
  • achieve an average grade of 50 or more in written assignments for those subjects;
  • achieve a grade of 45 or more in at least 8 of those written assignments;
  • attend a project week, and achieve a grade of 50 or more in the dissertation;
  • produce a satisfactory performance in the oral examination, unless dispensed from the requirement to attend one.

Students are expected to attend every morning and afternoon session of each teaching week. Students who are absent for more than two sessions will not normally be considered to have attended that course. In exceptional circumstances, with the approval of the Director of Graduate Studies, a longer period of absence may be allowed. Students should approach the Programme Office in the first instance.

To achieve a distinction for excellence, it is normally sufficient to also:

  • achieve an average grade of 70 or more in written assignments for the chosen subjects;
  • achieve a grade of 65 or more in at least 8 of those written assignments;
  • achieve a grade of 70 or more in the dissertation.

Should a student attend courses and submit assignments in more than 10 subjects, the Examiners will consider the 10 assignments that would best advantage the student in the examination for their chosen award.

Additional requirements apply to candidates for the MSc in Software and Systems Security:

  • the selection of assignment submissions considered by the examiners must include a numerical majority of security subjects (as defined by the degree schedule);
  • an average of 50 or more in the assignments in security subjects (in addition to having an overall assignment average of 50 or more) is necessary to pass the MSc;
  • an average of 70 or more in the assignments in security subjects (in addition to having an overall assignment average of 70 or more) is necessary to be awarded a distinction in the MSc;
  • the topic of the dissertation must have been approved as a topic in security. If this requirement is not satisfied, then students are advised to enter instead for the MSc in Software Engineering.

The results of examination will be published as soon as possible after the Examiners' meeting. Any candidate who fails to satisfy the examiners is given three terms in which to retrieve the situation: through the submission of additional written assignments, or the submission of a revised dissertation. This additional period of study begins with the start of the next term following the publication of results. No further extensions or intermissions are possible.

Candidates who have previously failed to satisfy the examiners in their intended award will not normally be considered for a distinction.

If a candidate fails to satisfy the examiners in respect of their intended award, but satisfies the examiners in respect of a lower award offered by the Programme, then they may be offered a choice between failure (and possible resubmission), and the award of the other qualification. The conventions set out conditions for these awards:

To be awarded a Postgraduate Diploma, it is normally sufficient to:

  • attend courses in 8 different subjects;
  • achieve an average grade of 50 or more in written assignments for those subjects;
  • achieve a grade of 45 or more in at least 6 of those written assignments;
  • produce a satisfactory performance in the oral examination, unless dispensed from the requirement to attend one.

To achieve a distinction for excellence, it is normally sufficient to also:

  • achieve an average grade of 70 or more in written assignments for the chosen subjects;
  • achieve a grade of 65 or more in at least 6 of those written assignments.

To be awarded a Postgraduate Certificate, it is normally sufficient to:

  • attend courses in 4 different subjects;
  • achieve an average grade of 50 or more in written assignments for those subjects;
  • achieve a grade of 45 or more in at least 3 of those written assignments;
  • produce a satisfactory performance in the oral examination, unless dispensed from the requirement to attend one.

To achieve a distinction for excellence, it is normally sufficient to also:

  • achieve an average grade of 70 or more in written assignments for the chosen subjects;
  • achieve a grade of 65 or more in at least 3 of those written assignments.

As at MSc level, the numerical majority of written assignments considered for awards in Software and Systems Security must be in security subjects (as defined by the degree schedule).

Plagiarism

Every submission to the Programme is preceded or accompanied by a declaration that it is the author's own work except where otherwise indicated. If part of the submission is based closely upon the work of another person, then this must be properly acknowledged: not to do so would be plagiarism, something that is seen as unacceptable in our academic culture. To help them avoid this, students are given guidance on the proper acknowledgement of sources (http://www.cs.ox.ac.uk/softeng/handbook/acknowledgement.html).

The University takes seriously the offence of plagiarism and its policy states:

No candidate shall present for an examination any part, or the substance of any part, of another person's work, as if it were the candidate's own work. Candidates must provide a declaration with every assignment that the work submitted is their own work. In any written work passages quoted or closely paraphrased from another person's work must be identified as quotations or paraphrases, and the source of the quoted or paraphrased material must be acknowledged.

'Examination' includes where the context so permits the submission and assessment of a thesis, dissertation, essay, practical work, or other coursework which is not undertaken in formal examination conditions but counts towards or constitutes the work for a degree or other academic award.

The examiners may impose a penalty of up to 10 marks for minor cases of poor academic practice (such as inadequate attribution of sources) over small proportions of the submitted work (not exceeding 10% of the whole). More serious or more extensive infractions will be referred to the Proctors. If the Proctors decide that there is a case to answer, the student will be formally charged with an offence and a university disciplinary process will follow. The outcome will depend on the severity of the case and may range from a reduction in marks to expulsion from the course. If the Proctors decide that there is no case to answer, the student will be referred back to the Department to be offered appropriate guidance and feedback on referencing and assignment writing.

The assignments for taught courses are examinations of the University of Oxford, as are the project and dissertation component of the MSc. The wider University Regulations regarding examinations (http://www.admin.ox.ac.uk/examregs/) apply to all registered students on the Programme.

Results and Feedback

If any student wishes to have clarification of their results, or the contents of the assessment report, they should contact their supervisor via the Programme Office.

If the student is not satisfied with the response from the Programme Office, then they may make a formal appeal against the decision by writing to the University Proctors via the Senior Tutor of their college.

The Proctors will ensure that all examinations are conducted in accordance with the University's Statutes and Regulations. They are not empowered to challenge properly exercised academic judgement.