Edit on GitHub

Grader.jl Documentation

Grader is an assignment autograder for Julia. It has been designed to be used with the PrairieLearn learning system, but it could also be used with any other learning system.

Examples

Simple example

Here is a simple example of how to use Grader:

using Grader 

# This is the code with the correct answer:
goldencode = """
x=2
y = x * 3
"""

# This is the code with the student's answer:
studentcode = """
x=2
y = x + x + x
"""

p = Problem()
golden = @rungolden! p goldencode
student = @runstudent! p studentcode

grade!(p, "y", "check y", 2, :($student.y ≈ $golden.y), "y is incorrect")
p

# output

Problem
  gradable: Bool true
  score: Float64 1.0
  message: String ""
  output: String ""
  images: Array{Grader.Image}((0,))
  tests: Array{Grader.TestResult}((1,))

Example with incorrect answer

Here's an example of what it looks like when the student gets the wrong answer:

using Grader

# This is the code with the correct answer:
goldencode = """
x=2
y = x * 3
"""

# This is the code with the student's answer:
studentcode = """
x=2
y = x + x + 2x
"""

p = Problem()
golden = @rungolden! p goldencode
student = @runstudent! p studentcode

grade!(p, "y", "check y", 2, :($student.y ≈ $golden.y), "y is incorrect")
p.tests[1]

# output
Grader.TestResult
  name: String "y"
  description: String "check y"
  points: Float64 0.0
  max_points: Float64 2.0
  message: String "y is incorrect"
  output: String ""
  images: Array{Grader.Image}((0,))

We can also forbid students from using certain symbols (e.g libraries):

using Grader

studentcode = """
using LinearAlgebra
x=2
y = x + x + 2x
"""

p = Problem()
@runstudent! p studentcode [:LinearAlgebra]

p.output[1:63]

# output
"error running student code:\nUsing LinearAlgebra is not allowed."

Example with plot

This is a more complex example that grades a plot and writes out the output as JSON.

using Grader
using LinearAlgebra, JSON

studentcode = """
using Plots
x=1:10
y = x.^2
xy = plot(x,y)
"""

p = Problem()
student = @runstudent! p studentcode

grade!(p, "XY Plot", "Data length", 1, 
    quote
        xlen = length($student.xy[1][1][:x])
        ylen = length($student.xy[1][1][:y])
        xlen == ylen == 10
    end, 
    "the number of data points in x or y is incorrect")

grade!(p, "XY Plot", "Y values", 3, 
    quote
        ynorm = $norm($student.xy[1][1][:y])
        ynorm ≈ 159.16343801262903
    end, 
    "The Y values are not correct")

# Output to JSON and print the result
b = IOBuffer()
pl_JSON(b, p)
JSON.print(JSON.parse(String(take!(b))), 4)

# output
{
    "images": [],
    "score": 1.0,
    "output": "",
    "message": "",
    "gradable": true,
    "tests": [
        {
            "images": [],
            "name": "XY Plot",
            "points": 1.0,
            "output": "",
            "message": "",
            "max_points": 1.0,
            "description": "Data length"
        },
        {
            "images": [],
            "name": "XY Plot",
            "points": 3.0,
            "output": "",
            "message": "",
            "max_points": 3.0,
            "description": "Y values"
        }
    ]
}

API Documentation

Grader.Problem — Type

Represent a problem for grading.

Fields:

  • Gradable: whether the problem is gradable, i.e. whether the all relavent code has executed without any errors
  • Score: the score of the problem, as a fraction of the maximum possible score
  • Message: Any message associated with the graded problem
  • Output: The output of the problem grading
  • Images: Any images associated with the problem grading
  • Tests: A list of tests associated with the problem
source
Grader.fill_answers — Method
fill_answers(code::AbstractString, answerkey::Dict)::String

Fill the answer key into a code template and return the resulting code string.

For example, consider the code template below, that gives the radius of a circle and asks the student to fill in the area and perimeter. In the template, the area and perimeter are given as variables a and p with values missing, and the student is meant to replace missing with the actual answer.

code = "# Calculate the area and perimeter of a circle with radius 2.\n"*
"r = 2\n"*
"a = missing # Area\n"*
"p = missing # Perimeter\n"

answer_code = fill_answers(code, Dict(
    :(a = missing) => :(a = π * r^2),
    :(p = missing) => :(p = 2π * r)));


p = Problem()
answer = @runstudent! p answer_code
grade!(p, "area", "calculate area", 1, :($answer.a ≈ 4π), "area is incorrect")
grade!(p, "perimeter", "calculate perimeter", 1, :($answer.p ≈ 4π), "perimeter is incorrect")

println("Answer key score is $(p.score).")

# output
Answer key score is 1.0.
source
Grader.grade! — Method
grade!(p::Problem, name::String, description::String, points::Real, expr::Expr, msg_if_incorrect::String)

Add a grade for problem p. This grade will have the given name and description in the grader output, and will be associated with the number of points.

The function will evaluate the given expression expr; if it evaluates to true, the given number of points will be awarded, otherwise zero points will be awarded and msg_if_incorrect will be logged to the problem p.

source
Grader.@rungolden! — Macro
rungolden! p::Problem goldencode::AbstractString

Run the provided code string inside a module and return the module. If an error occurs it will be logged in Problem p as a problem with the "golden" code.

source
Grader.@runstudent! — Macro
@runstudent! p::Problem studentcode::AbstractString

Run the provided code string inside a module and return the module. If an error occurs it will be logged in Problem p as a problem with the "student" code.

source