Marking Scheme Generator - nns@aber.ac.uk - 17 Oct 2010

Contents

Overview

The software provides support for structured marking of student coursework to enable detailed feedback and overall marks to be produced with minimum effort.  The concept is that a marking scheme template is created containing assignments, tasks, and grades.  The course tutor describes what is required to obtain each grade with comments explaining the aspects that can be improved.  This allows feedback to be automatically generated based on the grades selected during the marking process, combined with any bespoke comments that are required.  The software can then assemble and email the individual feedback for each student based on the grades, feedback options, and bespoke feedback created. 

Cohort information (student reference, names, email) can be imported from spreadsheet or  .csv files  and conventional 'blank' template spreadsheets can be produced from a marking scheme or completed spredsheets produced if grades are entered into the tool.


There is functionality remove all student related information from a marking scheme to create a new template for a subsequent cohort or assignment.  The marking scheme can be modified (as far as is reasonable) as marking proceeds if the need arises. 

The software provides main functions as follows and outlined in the diagram below.



Architecture diagram
Marking schemes are decomposed into several levels.  The following list describes the characteristics of each level:
In all cases the aim the to provide feedback that will provide pointers as to how the student could improve (if necesary) the aspect under consideration. 

Student lists may be imported from Excel (.xls, .xlsx) or comma separated text files (.csv) into a marking scheme to allow a marking/results spreadsheet to be generated for the cohort.  This spreadsheet provides one way that grades and marks can be entered during marking.  Once marking is complete all necessary calculations are completed by the spreadsheet to produce the final module mark for each student. A set of feedback will also be generated for each task ready for distribution to individual students.

The system architecture is based around capture of the marking scheme in an xml file generated by a GUI tool.  A list of students is added to allow marking to proceed and the marking spreadsheet to be generated for a specific cohort.  The spreadsheet can be used to enter grades allowing simple grade based feedback to be generated on the spreadsheet, however the more sophisticated and flexible feedback generation options are only available on the GUI version. The marking scheme can be saved at any time as a template that may then be used for a subsequent cohort.

An information area at the bottom of the tool will provide confirmation of important actions and reports of any problems. It also provides an overview numerical contributions of the various marking scheme parts.

The Marking Scheme Generator GUI is comprised of several tabs that deal with the main functions of the system:

Running the tool

The tool is distributed as a '.jar' java archive file and on recent versions of most operating systems may be executed by double clicking on the markingsheet.jar file.  If this fails the tool may be run from the command line using one of the following commands as long as the system paths are set such that the '.jar' file can be found:

java -jar markingsheet.jar
java -jar markingsheet.jar path_and_filename.xml

The first form will open the tool with an empty marking scheme, the second will open the marking scheme contained in the file 'filename'.  If the filename includes a path (e.g ../module/marking/filename.xml) then this will become the default folder for subsequent file operations.

The tool requires a Java runtime environment (JRE) version 1.6 or higher.

Loading an existing marking scheme

An existing marking scheme can be loaded by clicking on the 'Load existing marking scheme' button.  A File chooser will then appear unless a marking scheme name has been entered in the 'marking scheme name' box AND that file is in the working directory.  The working directory will be set when a file is loaded using the file chooser or will be the directory the tool was run from.  Marking schemes are xml files and will often appear with an 'xml' suffix on many platforms.

There are some example schemes in the 'examples' directory of the installation.

Marking Schemes

A marking scheme is generated from the Module and Assessments tabs.

The Module page allows entry of the module overview information, together with loading and saving buttons.  The Assessment tab allows the details of assesments, tasks, grades and weightings to be entered.

Module Tab

The Module page allows entry of the module overview information, together with loading and saving buttons.  The following tables describe what should be entered on the form.

Module The formal module identifier should be entered
Title The descriptive module title should be entered
Marking scheme
A name should be provided for the marking scheme to be used as part of the file name.  This could be the module code or condenced title if desired.
Version
A number stored as part of the marking scheme for future reference purposes
File containing students
The name of the spreadsheet (.xls, .xlsx, .csv) file containing the student details for a  specific cohort can be provided.
Year
A cohort year must be entered and will form part of the marking scheme filename. A format of "2011-2012" is suggested

A module marking scheme may be saved as a template (i.e. no students included) as soon as the above information is provided.

Loading and saving marking schemes.

A marking scheme without student data is saved using the 'Save marking template' button.  Once students have been included, the marking scheme is saved using the 'Save marking scheme' button.  A template for use with a new cohort (using the same marking scheme) can be created using the 'Save as template for new cohort' button.

Load existing marking scheme
If a marking scheme name has been provided (.xml extension not required) and it exists in the working directory it will be loaded. Otherwise a file chooser will appear to allow navigation to the required marking scheme file.
Create blank marking scheme
A new marking scheme is created in place of any that is loaded.  Ensure any existing marking scheme is saved first.
Save as template for new cohort
A new marking scheme template will provide a file chooser to allow navigation to the folder in which to store the new marking scheme.  Subsequently, and for templates (or marking schemes) that were obtained from files the existing file will be updated.
Load student list
If the student file can be found in the working directory it is loaded.  Otherwise a file chooser will appear to allow navigation to a suitable spreadsheet (.xls) file. The file name will then appear in the student file field. The structure of the file should be specified using the appropriate row and column numbers presented.
Save marking scheme
The marking scheme is saved into a file with a name comprised of the marking scheme name and the cohort year.  A new marking scheme will provide a file chooser to allow navigation to the folder in which to store the new marking scheme.  Subsequently, and for templates (or marking schemes) that were obtained from files the existing file will be updated.

Loading a Cohort

Student lists can be extracted from spreadsheet or comma separated text files (.xls, .xlsx, .csv ) and must provide the student ID, Name and Email.  It is important to specify the rows and columns in which the data is to be found using the four fields provided.  The student ID, name and email values specify the column (or nth item from the start of the row in the case of .csv).  The first row that contains valid student data should also be specified since many files and spreadsheets contain header rows that should be skipped.  All other students must occupy adjacent rows or lines in the case of a .csv file.  All additional columns in the sheet or file are ignored. The first row and column in the spreadsheet or file is numbered 1. Take special care to check that the first and last student have been included in case that the input spreadsheet has differering numbers of header rows to what is expected. 

Adjusting the Cohort

If the cohort editing functions are enabled (see properties) the add/delete and reorder buttons appear under the cohort list on the module tab.  The ID, name and email for each student can be edited when selected in the list using the edit area.  Additional students can be added, however it is not intended that an entire cohort be entered in this way; it should be imported from a list exported from the student records system.  In particular the student IDs must be correct since all information, marks, feedback, and spreadsheets are indexed and referenced using this unique number.  Changing a students ID will result in any marks being lost for that student.

If a student is removed from the cohort list the ID will remain in all assignment/task lists but the student will not appear on any spreadsheets or the results/feedback tabs.   Adding the student (with the same ID) will result in those marks etc becoming available again.

Assessment Tab

A module is considered as a set of assessments (i.e. assignments and exams), each contributing a percentage of the overall module mark. Each assignment is considered as a set of tasks (questions in the case of exams) that each contribute a percentage of the mark for an assesment.  The task percentages for an assignment should sum to 100%. The sum of all assesments (including exams) should also total 100%.  These calculations will be build in to the spreadsheet produced.

A choice between a set of qualitative grades and numerical mark can be made.  Numerical marks are required for exam questions.  For assignments qualitative grades should be used each being mapped to a numerical mark.  The grade providing the maximum mark is considered as 100% for the task.  It may be advantaegous for the purpose of feedback to provide more than one grade that maps to a specific mark.  These grades may indicate alternative ways that a student might achieve a common mark, allowing alternative feedback to be  provided.

The following annotated diagram illustrates the main points associated with creating a series of assessments:


The Assessment, Task and Grade names will appear on the generated spreadsheet as column titles or row entries and should preferebly be relatively concise.  The description and feedback elements allow verbose multi line descriptions of these elements.  The assessment name in particular is used in the name of sheets within any generated workbook and Excel limitations mean truncation will occur if the assessment name is longer than 20 characters.

Marking - Results Tab

The Results tab provides facilities for entering grades and marks. The student and assessment are selected from lists and the required grade is selected for each task from the drop down lists as required. There is a check box used to indicate non submission.  As each grade is selected a preview will be shown in the message area at the bottom of the window.

Optional Feedback

The 'view optional feedback' button will open a pane that allows additional feedback options associated with grades to be selected.  These may have been generated using the 'Optional Feedback' pane from the assessments tab.  It is possible to select any combination of optional feedback associated with a grade.  Feedback from other grades may also be utilised by clicking the 'Show feedback for all grades' button.  Once an item is selected it will become part of the optional feedback for the selected grade. Additional items of optional feedback can be added on the Assessments tab as required.  Grade based feedback options associated with other grades will appear shaded util added to the currently selected grade.

If required bespoke feedback can be added for a task.  This feedback can be reused for other students by pressing the 'Show all task feedback' selecting it. Bespoke feedback from other students is shown shaded until selected.

Groups


The system is able to show all of the students that have identical feedback for an assignment.  These are called groups.  Note that a group does not link students in any way and as soon as any feedback element is changed a student will no longer be part of the group it belonged to before the change. The student may become part of another group however.  Using the 'View groups' button will open a new pane that shows all students with a checkbox to indicate those that are in the same group as the currently selected student on the results window.

Clicking on a student name causes all differences in marks and feedback between this student and the selected one to appear in the messages area of the main window. 

Clicking on an unchecked checkbox for a student will copy all marks and all feedback from the student selected on the results pane.  There will be a warning is this will cause any existing feedback or marks to be replaced.  The details of these replacements are shown in the message area in red. The items that will be copied will be in black.

Remember that if the feedback is subsequently changed for a student in a group it will no longer be part of the group - the checkboxes can be rechecked to recopy the feedback if required.

Generating Spreadsheets

A marking spreadsheet can be generated once a cohort of student details have been included in the marking scheme.  The spreadsheet contains all the formulas and look up tables required to produce the module results once the grades for each student are entered. These grades are entered by selecting the appropriate grade from a list of valid grades presented when cells are selected.  In addition the spreadsheet bases all lookups and transfer of information between sheets on the student ID.  This protects the spreadsheet against modifications to student lists etc.  It is possible to add to the grades, however there are a couple of points to consider to ensure full functionality such as drop down lists maintained. 

The Spreadsheets tab is used to generate the spreadsheet. It will be given the same name as the marking scheme but will have the '.xlxs' extension and may be opened by Microsoft Excel or any compatable program.

Spreadsheets Tab

As far as possible all spreadsheet rows, columns and cells will be automatically sized to fit their contents.  There a few situations where this is not possible, for example the desired width for a cell that wraps its (potentially verbose) contents.  The spreadsheets tab therefore allows preferences for these parameters to be specified.  A default set of reasonable values are provided when a new marking scheme is created which should be adequate for most situations.  Check the generated spreadsheet for good layout before filling in marks, because it is far less time consuming to regenerate the spreadsheet if the layout is not as desired than changing all of the sheets individually.

Parameter
Description
Default
Feedback column width
The number of character columns allocated for cells containing the feedback comments
20
Individual comments column width
The number of character columns allocated for cells that will contain free form comments for an assessment.
35
Marks columns width
The width of any columns containing numerical marks
7
Student ID column width
The width of the student ID column (this can't be automatically calculated due to an Excel technicality because student ID cells are treated as a formula type to preserve leading zeros and also ).
15
Grade description height (pts)
The grade descriptions are given more space for aesthetic reasons and the vertical hight is specified in points.
40
Default sheet magnification (%)
The sheets that require grades input are given a default magnification to assist in the entry of grades. Set this to 100 for the default view.
125

Clicking the Create marking spreadsheet button will generate the spreadsheet sheet.  If the marking scheme contains problems that prevent the sheet being generated (such as zero percentage weighting, or empty task lists etc) these will be explained in the scrollable information area.

Spreadsheet Workbook Description

The spreadsheet will contain a 'Summary' worksheet, 'Results' worksheet plus four additional worksheets for each  assessment. The 'results' sheet contains the overall result for each assignment weighted according to its contribution to the module.  The summary sheet is similar except that it contains the task (ie question) marks for assignments with the text "exam" (case insensitive) anywhere in the title. The summary sheet obtains its 'raw mark' from the 'results' sheet and all other marks from the assignment/exam 'marks' sheet.  The four sheets for eaxh assignments/exam are:
Of the four assignment sheets it is usually it is only necessary to fill in the marks picking sheet for each assessment (see example below), and then to consider the results summary sheet.  The other sheets are accessible in case the marking scheme needs to be adjusted part way through marking, and so that the numerical marks generation process can be traced in case of exceptional circumstances. While it is possible to navigate between the required sheets using the sheet tabs it is far easier to use the named regions that have been created.  These can be accessed from the list selector that usually shows the name of any currently selected cell (e.g. 'A1') near the top left of the Formula bar (dependent on platform).  The formula bar may be hidden and there are various ways to make it appear dependent on the version and operating system used in MS Excel some suggestions are:
There will a number of entries, but the those of the form '__Assignment_name__' (Assignment_name replaced with names from the marking scheme) will go directly to the area of the page that requires marks to be entered for each assignment as in the example below.  The '__RESULTS__' can be used to go to the results summary sheet.  Don't forget to note any student specifc feedback that is not covered by the feedback associated with the selected grade in the Individual Comments column!


Propagation of data between sheets

For completeness, the diagram below outlines the propagation of data between the sheets of the workbook. The 'Results' and 'Summary' sheets derive their data from the 'Marks' sheet for each assignment (exam).  The 'Marks input sheet' can be generated to reorder the students and generate checksums for the purpose of data entry to the student record system; all of its data is obtained from the Summary sheet.



There are a few points worth noting should the need arise to modify marks other than using the 'Picking' sheets.
  1. Overriding the assignment marks on the 'Results' sheet will not be propagated to the 'Summary' sheet - best to override genuine errors on the 'Picking' or 'Marks' sheet and allow the result to propagate.  Circumstances adjustments to assignments or exam results best made on the 'Summary' sheet.
  2. Overriding the 'Raw' mark on the 'Results' sheet will result in the value being propagated to the 'Summary' sheet.  This will clearly also override any task marks it would originally have been calculated from. If a rescaling of marks for an entire cohort is required then it can be performed on the 'results' sheet and the final results placed in the 'Raw Mark' column. 
  3. Changing marks on the assessments '?_Marks' sheets will propagate to the 'Results' and 'Summary' sheet as long as the mark has not been overridden on these sheets. Hence mark scaling for an entire assignment can be carried out on the '?_Marks'sheet if required, with the required results placed in the 'Module Contribution %' mark column.
  4. Overriding any entry on the 'Marks Input' sheet will not cause the totals to be updated since all columns are derived directly from the 'Summary' sheet with no calculated columns other than the checksum.  

Non numerical entries in exams

The picking sheet will contain empty text values to be entered in all cells that do not have a mark allocated.  The empty text values will be propagated to other sheets, and will not participate in any calculations.  If is is necessary to enter marks that violate the exam rubric they will appear as the text value "! n" where n is the mark.  This value will also be propagated to the summary sheet, and will not participate in any calculations. The GUI requires that a violating mark is entered as "-n".

Named regions

For completeness the following information is provided. Each generated workbook will have several named regions.

__RESULTS__

For each assignment there will be:
__Assignment_name__  The input area of the picking sheet for an assignment. This is the area that must be completed by a marker.

The following areas are used to perform the necessary calculations, and the marker will not normally not need to use them.

Assignment_name_PICKING A larger area of the picking sheet for an assignment
Assignment_name_MARKS The numerical marks sheet, generated by lookup of the grades on the picking sheet using the grades sheet
Assignment_name_FEEDBACK The feedback sheet for an assignment.

For each task there will be:
Assignment_name_Task_name The grade, mark, and feedback for a task on the grades sheet of an assignment.
Assignment_name_Task_name_GRADES mark for a task on the grades sheet of an assignment.

Adding Grades to the spreadsheet

The grades are found in the Assignment_name_Task_name_GRADES named region of the workbook.

Add another grade level once the spreadsheet has been used:

Normally an additional grade can be inserted by:
  1. Selecting the name, grade and feedback columns in the row below the point in which the additional grade is to be inserted.
  2. Select Insert->Cells
  3. Choose 'Shift cells down'
  4. Type the additional task into the newly created cells
  5. The new grade will appear on the grade picking sheet selection menus.
  6. If the mark range is extended then the appropriate cell in the 'maximum available mark' row on the '*_Marks' sheet should be adjusted accordingly (otherwise the new grade will result in >100%).
If for some reason the new grades do not appear on the mark picking sheet then the most likely cause is that the named region that refers to the task required has become corrupted.  This can be corrected as follows:
  1. Redefine the label for that task.
  2. Insert->Name->define
  3. This willproduce a window with the list of names in it
  4. Find the two items in the list for the task concerned
  5. Change the very last number (cell row reference) in the `Refers to': box to the last row containing the added items.  If only one is added simply increment it.

Adding Students to the spreadsheet

Insert another student in the results and/or other sheets once the spreadsheet has been used:
  1. Click the row number of the student after the one to be inserted.
  2. Insert->Row
  3. Copy any student row and paste it into the new row.
  4. Cdit the student ID and name and email to the required values.
This must be done on each sheet for which the student has completed an assessment item. On the results sheet any missing assessment items will appear as `#N/A' simply fill in the cell with the required value (i.e. `0').

Anonymous marking

If the anonymous marking function is enabled (see properties) the tickbox under the 'individual feeedback' area on the results tab may be used to remove the student name from the results entry tab.  All students are then identified by their student reference number ('ID') .  To enable  the correct student to be rapidly found the student ID can be entered into the 'instant search' area above the cohort list; as soon as a unique ID matching the charactered entered is found the student will be selected automatically.  Check the remainder if the ID matches the script being marked !

Auto Backups

If the auto backup function is enabled (see properties) the MSG will save a copy of the current marking scheme in the same directory as the main file copy is being stored but with the additional extension that includes the time and date the program was run.  There is therefore one backup file created each time the MSG is run.  The file is updated each time a new student is selected, each time a switch is made between tabs, and when the system is exited.  On most systems this does not have any noticable effect on performance (the files are small) and allows recovery of data up to the last result entered in event of a power failure or system crash. This file can have the extension removed following the '.xml' and be loaded directly into the MSG. Ensure the backup is more recent than the normally saved version though! When working on very slow devices the function can be disabled if changing between students or tabs is too sluggish.  The reasoning behind saving a new file for each execution of the MSG is to avoid a good backup being overwritten by executing the program again before it is realised that entered data was not saved for some reason. 

The backup file will only be saved when a valid marking scheme is available (data entered conforms to the schema) and no failure messages are reported if it fails (no disk space, locked directory etc. etc).  Do not use it as a substitute for regularly using the save functions since these will report any problems in the information area.

Properties File

The first time the system is run the file 'MSG Properties' will be created in the directory containing the executable.  When the MSG is run on future occasions the file will be used to configure the MSG if it exists. To return to the default settings simply delete the file. The 'MSG Properties' file contains contains a variety of configurations for the MSG in simple text format.  Many are associated with the default sizes and colours of the graphical interface elements, however a few may be usefully modified:

Feature enable/disable (disabling options can be useful on lower resolution displays to same space for example):
Local configurations
Layout

Future Enhancements

The following are possibilities for future enhancement.  Suggestions to nns@aber.ac.uk.

Optional tasks

Some tasks (exam questions in particular) may be optional.  The current schema does not support choices between tasks or assignments.  The spreadsheet cannot therefore enforce such constraints.  If an exam has a rubric Answer X from Y questions, it is possible to provide marks for Y questions, presumably creating a mark of greater than 100% for the exam for that student.  This possibility is indicated in the information panel for the assessment which will also show a potential assignment percentage of greater than 100%.  

When entering marks selective tasks only the marks that are to count should be entered into the task columns, dependent on the rules to be applied for not obeying the rubric. In the above illustration this typically means the X answered questions with the highest mark. A note of the marks for discounted elements can be noted in the "individual feedback' column, which for exams is not otherwise normally used.  Marks can be entered for all the tasks if required, and of course could total more than 100%.


Enjoy your marking :)
Neal.