Requirements

The purpose of the archive is to capture, retain and provide controlled access to the above-mentioned information.

The most important near-term need is to start capturing the data. Ultimately, there will be a lot more access than capture, but capture must precede access, and the accessing will be relatively simple once the captured data is properly indexed. The accesses that need rapid response will be humans looking up of archived information. Long-running scripts will do statistical analyses, but those scripts will not be run so frequently and can take a long time to do their work.

The Archive must provide script-generated reports that facilitate the continuous-improvement process and the necessary files for ABET evaluations:^2.5

• course files, which contain everything relevant to a given course offering, including three samples (a high, a medium, and a low) of graded work for each assignment, quiz, etc.
• per-outcome files that track the degree to which our graduates achieve each targeted outcome.

We also want to facilitate and standardize the generation of Department-to-student transactions, e.g., the generation of exams of a desired difficulty and purpose.

The Archive will support a technique called ``course-based assessment'' (CBA) to determine on a per-student basis whether or not a given performance criterion has been met.^2.6Under CBA, instructors for various courses are responsible for including questions on their exams and problems in their assignments that assist in determining whether students have achieved met a particular performance criterion.

The technique we intend to use might be better called ``assessment-based assessment'' -- we intend to use the normal process of assessing student work to assess how well we are teaching them. To do a better job of assessing the students (and by implication, our own instructional efforts), we will keep a database of how each student performs on each question s/he attempts throughout the Department. The points being that:

• The best way to assess the effectiveness of our instruction is to frequently assess the competencies of the students and analyze the data.^2.7
• As the questions assess the students, the students are implicitly assessing the questions; the scores on a given question tell as much about the question as they do about the students who took it.

To that data, we will apply techniques from assessment theory -- see [R28], [R29], and [R30]. It turns out that such computer-aided assessment (CAA) is big business in Britain -- see for example http://www.itembank.co.uk/.

We intend to accomplish the above by keeping everything that's relevant on-line in places and formats that scripts can find and parse and thus generate the above-mentioned reports, plus the on-going feedback reports to support our continuous-improvement process.

Although we have until the end of Fall '03 before the first reports must be generated, by the beginning of Fall '03, we need to tell the faculty, TAs, and graders what to save, and where, how, and in what format to save stuff. Of course, we can ask them to submit each item to a script that will put it in the right place and format. But, the script(s) must be working by the beginning of the quarter, and their interfaces must be friendly enough that they will be used. Also, we'll need some mechanism by which to check that stuff is being properly saved.