Programming Exercise


Overview

Conducting a programming exercise consists of 7 steps distributed among instructor, Artemis and students:

  1. Instructor prepares exercise: Set up a repository containing the exercise code and test cases, build instructions on the CI server, and configures the exercise in Artemis.

  2. Student starts exercise: Click on start exercise on Artemis which automatically generates a copy of the repository with the exercise code and configures a build plan accordingly.

  3. Optional: Student clones repository: Clone the personalized repository from the remote VCS to the local machine.

  4. Student solves exercise: Solve the exercise with an IDE of choice on the local computer or in the online editor.

  5. Student uploads solution: Upload changes of the source code to the VCS by committing and pushing them to the remote server (or by clicking submit in the online editor).

  6. CI server verifies solution: verify the student’s submission by executing the test cases (see step 1) and provide feedback which parts are correct or wrong.

  7. Student reviews personal result: Reviews build result and feedback using Artemis. In case of a failed build, reattempt to solve the exercise (step 4).

  8. Instructor reviews course results: Review overall results of all students, and react to common errors and problems.

The following activity diagram shows this exercise workflow.

Exercise Workflow

Exercise Workflow

Setup

The following sections describe the supported features and the process of creating a new programming exercise.

Features

Artemis and its version control and continuous integration infrastructure is independent of the programming language and thus supports teaching and learning with any programming language that can be compiled and tested on the command line. Instructors have a lot of freedom in defining the environment (e.g. using build agents and Docker images) in which student code is executed and tested. To simplify the setup of programming exercises, Artemis supports several templates that show how the setup works. Instructors can still use those templates to generate programming exercises and then adapt and customize the settings in the repositories and build plans.

  • The support for a specific programming language templates depends on the used continuous integration system. The table below gives an overview:

    Programming Language

    Bamboo

    Jenkins

    Java

    true

    true

    Python

    true

    true

    C

    true

    true

    Haskell

    true

    true

    Kotlin

    true

    false

    VHDL

    true

    false

    Assembler

    true

    false

    Swift

    true

    false

  • Not all templates support the same feature set. Depending on the feature set, some options might not be available during the creation of the programming exercise. The table below provides an overview of the supported features:

    Programming Language

    Sequential Test Runs

    Static Code Analysis

    Plagiarism Check

    Package Name

    Solution Repository Checkout

    Java

    true

    true

    true

    true

    false

    Python

    true

    false

    true

    false

    false

    C

    false

    false

    true

    false

    false

    Haskell

    true

    false

    false

    false

    true

    Kotlin

    true

    false

    false

    true

    false

    VHDL

    false

    false

    false

    false

    false

    Assembler

    false

    false

    false

    false

    false

    Swift

    false

    false

    false

    false

    false

    • Sequential Test Runs: Artemis can generate a build plan which first executes structural and then behavioral tests. This feature can help students to better concentrate on the immediate challenge at hand.

    • Static Code Analysis: Artemis can generate a build plan which additionally executes static code analysis tools. Artemis categorizes the found issues and provides them as feedback for the students. This feature makes students aware of code quality issues in their submissions.

    • Plagiarism Checks: Artemis is able to automatically calculate the similarity between student submissions. A side-by-side view of similar submissions is available to confirm the plagiarism suspicion.

    • Package Name: A package name has to be provided

    • Solution Repository Checkout: Instructors are able to compare a student submission against a sample solution in the solution repository

Note

Only some templates for Bamboo support Sequential Test Runs at the moment.

Note

Instructors are still able to extend the generated programming exercises with additional features that are not available in one specific template.

We encourage instructors to contribute improvements to the existing templates or to provide new templates. Please contact Stephan Krusche and/or create Pull Requests in the Github repository.

Exercise Creation

  1. Open Course Management

  • Open course-management

  • Navigate into Exercises of your preferred course

    ../../../_images/course-management-course-dashboard.png
  1. Generate programming exercise

  • Click on Generate new programming exercise

    ../../../_images/course-management-exercise-dashboard.png
  • Fill out all mandatory values and click on generate

    ../../../_images/create-programming-1.png
    ../../../_images/create-programming-2.png

    Result: Programming Exercise

    ../../../_images/course-dashboard-exercise-programming.png

    Artemis creates the repositories:

    • Template: template code, can be empty, all students receive this code at the beginning of the exercises

    • Test: contains all test cases, e.g. based on JUnit, hidden for students

    • Solution: solution code, typically hidden for students, can be made available after the exercise

    Artemis creates two build plans

    • Template: also called BASE, basic configuration for the test + template repository, used to create student build plans

    • Solution: also called SOLUTION, configuration for the test + solution repository, used to manage test cases and to verify the exercise configuration

    ../../../_images/programming-view-1.png
    ../../../_images/programming-view-2.png
    ../../../_images/programming-view-3.png
  1. Update exercise code in repositories

  • Alternative 1: Clone the 3 repositories and adapt the code on your local computer in your preferred development environment (e.g. Eclipse).

    • To execute tests, copy the template (or solution) code into a folder assignment in the test repository and execute the tests (e.g. using maven clean test)

    • Commit and push your changes submit

    • Notes for Haskell: In addition to the assignment folder, the executables of the build file expect the solution repository checked out in the solution subdirectory of the test folder and also allow for a template subdirectory to easily test the template on your local machine. You can use the following script to conveniently checkout an exercise and create the right folder structure:

      #!/bin/sh
      # Arguments:
      # $1: exercise short name as specified on Artemis
      # $2: (optional) output folder name
      #
      # Note: you might want to adapt the `BASE` variable below according to your needs
      
      if [ -z "$1" ]; then
        echo "No exercise short name supplied."
        exit 1
      fi
      
      EXERCISE="$1"
      
      if [ -z "$2" ]; then
        # use the exercise name if no output folder name is specified
        NAME="$1"
      else
        NAME="$2"
      fi
      
      # default base URL to repositories; change this according to your needs
      BASE="ssh://git@bitbucket.ase.in.tum.de:7999/$EXERCISE/$EXERCISE"
      
      # clone the test repository
      git clone "$BASE-tests.git" "$NAME" && \
        # clone the template repository
        git clone "$BASE-exercise.git" "$NAME/template" && \
        # clone the solution repository
        git clone "$BASE-solution.git" "$NAME/solution" && \
        # create an assignment folder from the template repository
        cp -R "$NAME/template" "$NAME/assignment" && \
        # remove the .git folder from the assignment folder
        rm -r "$NAME/assignment/.git/"
      
  • Alternative 2: Open edit-in-editor in Artemis (in the browser) and adapt the code in online code editor

    • You can change between the different repos and submit the code when needed

  • Alternative 3: Use IntelliJ with the Orion plugin and change the code directly in IntelliJ

    Edit in Editor

    ../../../_images/instructor-editor.png
  • Check the results of the template and the solution build plan

  • They should not have the status build_failed

  • In case of a build_failed result, some configuration is wrong, please check the build errors on the corresponding build plan.

  • Hints: Test cases should only reference code, that is available in the template repository. In case this is not possible, please try out the option Sequential Test Runs

  1. Optional: Adapt the build plans

  • The build plans are preconfigured and typically do not need to be adapted

  • However, if you have additional build steps or different configurations, you can adapt the BASE and SOLUTION build plan as needed

  • When students start the programming exercise, the current version of the BASE build plan will be copied. All changes in the configuration will be considered

  1. Adapt the interactive problem statement

../../../_images/course-dashboard-programming-edit.png
  • Click the edit button of the programming exercise or navigate into edit-in-editor and adapt the interactive problem statement.

  • The initial example shows how to integrate tasks, link tests and integrate interactive UML diagrams

  1. Configure Grading

../../../_images/configure-grading.png
  1. Verify the exercise configuration

  • Open the view page of the programming exercise

    ../../../_images/solution-template-result.png
  • The template result should have a score of 0% with 0 of X passed

  • The solution result should have a score of 100% with X of X passed

  • Click on edit

    • Below the problem statement, you should see Test cases ok and Hints ok

    ../../../_images/programming-edit-status.png

Online Editor

The following screenshot shows the online code editor with interactive and dynamic exercise instructions on the right side. Tasks and UML diagram elements are referenced by test cases and update their color from red to green after students submit a new version and all test cases associated with a task or diagram element pass. This allows the students to immediately recognize which tasks are already fulfilled and is particularly helpful for programming beginners.

Online Editor

Online Editor

Testing with Artemis Java Test Sandbox

Artemis Java Test Sandbox (abbr. AJTS) is a JUnit 5 extension for easy and secure Java testing on Artemis.

Its main features are

  • a security manager to prevent students crashing the tests or cheating

  • more robust tests and builds due to limits on time, threads and io

  • support for public and hidden Artemis tests, where hidden ones obey a custom deadline

  • utilities for improved feedback in Artemis like processing multiline error messages or pointing to a possible location that caused an Exception

  • utilities to test exercises using System.out and System.in comfortably

For more information see https://github.com/ls1intum/artemis-java-test-sandbox