Programming Exercise

Overview

Conducting a programming exercise consists of 7 steps distributed among instructor, Artemis and students:

1. Instructor prepares exercise: Set up a repository containing the exercise code and test cases, build instructions on the CI server, and configures the exercise in Artemis.

2. Student starts exercise: Click on start exercise on Artemis which automatically generates a copy of the repository with the exercise code and configures a build plan accordingly.

3. Optional: Student clones repository: Clone the personalized repository from the remote VCS to the local machine.

4. Student solves exercise: Solve the exercise with an IDE of choice on the local computer or in the online editor.

5. Student uploads solution: Upload changes of the source code to the VCS by committing and pushing them to the remote server (or by clicking submit in the online editor).

6. CI server verifies solution: verify the student’s submission by executing the test cases (see step 1) and provide feedback which parts are correct or wrong.

7. Student reviews personal result: Reviews build result and feedback using Artemis. In case of a failed build, reattempt to solve the exercise (step 4).

8. Instructor reviews course results: Review overall results of all students, and react to common errors and problems.

The following activity diagram shows this exercise workflow.

Exercise Workflow

Setup

The following sections describe the supported features and the process of creating a new programming exercise.

Features

Artemis and its version control and continuous integration infrastructure is independent of the programming language and thus supports teaching and learning with any programming language that can be compiled and tested on the command line. Instructors have a lot of freedom in defining the environment (e.g. using build agents and Docker images) in which student code is executed and tested. To simplify the setup of programming exercises, Artemis supports several templates that show how the setup works. Instructors can still use those templates to generate programming exercises and then adapt and customize the settings in the repositories and build plans.

• The support for a specific programming language templates depends on the used continuous integration system. The table below gives an overview:

  Programming Language	Bamboo	Jenkins
  Java	true	true
  Python	true	true
  C	true	true
  Haskell	true	true
  Kotlin	true	false
  VHDL	true	false
  Assembler	true	false
  Swift	true	false

• Not all templates support the same feature set. Depending on the feature set, some options might not be available during the creation of the programming exercise. The table below provides an overview of the supported features:

  Programming Language	Sequential Test Runs	Static Code Analysis	Plagiarism Check	Package Name	Solution Repository Checkout
  Java	true	true	true	true	false
  Python	true	false	true	false	false
  C	false	false	true	false	false
  Haskell	true	false	false	false	true
  Kotlin	true	false	false	true	false
  VHDL	false	false	false	false	false
  Assembler	false	false	false	false	false
  Swift	false	false	false	false	false

  • Sequential Test Runs: Artemis can generate a build plan which first executes structural and then behavioral tests. This feature can help students to better concentrate on the immediate challenge at hand.

  • Static Code Analysis: Artemis can generate a build plan which additionally executes static code analysis tools. Artemis categorizes the found issues and provides them as feedback for the students. This feature makes students aware of code quality issues in their submissions.

  • Plagiarism Checks: Artemis is able to automatically calculate the similarity between student submissions. A side-by-side view of similar submissions is available to confirm the plagiarism suspicion.

  • Package Name: A package name has to be provided

  • Solution Repository Checkout: Instructors are able to compare a student submission against a sample solution in the solution repository

Note

Only some templates for Bamboo support Sequential Test Runs at the moment.

Note

Instructors are still able to extend the generated programming exercises with additional features that are not available in one specific template.

We encourage instructors to contribute improvements to the existing templates or to provide new templates. Please contact Stephan Krusche and/or create Pull Requests in the Github repository.

Exercise Creation

1. Open Course Management

• Open

• Navigate into Exercises of your preferred course

  

2. Generate programming exercise

• Click on Generate new programming exercise

  

• Fill out all mandatory values and click on

  

  

  Result: Programming Exercise

  

  Artemis creates the repositories:

  • Template: template code, can be empty, all students receive this code at the beginning of the exercises

  • Test: contains all test cases, e.g. based on JUnit, hidden for students

  • Solution: solution code, typically hidden for students, can be made available after the exercise

  Artemis creates two build plans

  • Template: also called BASE, basic configuration for the test + template repository, used to create student build plans

  • Solution: also called SOLUTION, configuration for the test + solution repository, used to manage test cases and to verify the exercise configuration

  
  
  

3. Update exercise code in repositories

• Alternative 1: Clone the 3 repositories and adapt the code on your local computer in your preferred development environment (e.g. Eclipse).

  • To execute tests, copy the template (or solution) code into a folder assignment in the test repository and execute the tests (e.g. using maven clean test)

  • Commit and push your changes

  • Notes for Haskell: In addition to the assignment folder, the executables of the build file expect the solution repository checked out in the solution subdirectory of the test folder and also allow for a template subdirectory to easily test the template on your local machine. You can use the following script to conveniently checkout an exercise and create the right folder structure:

    #!/bin/sh
    # Arguments:
    # $1: exercise short name as specified on Artemis
    # $2: (optional) output folder name
    #
    # Note: you might want to adapt the `BASE` variable below according to your needs
    
    if [ -z "$1" ]; then
      echo "No exercise short name supplied."
      exit 1
    fi
    
    EXERCISE="$1"
    
    if [ -z "$2" ]; then
      # use the exercise name if no output folder name is specified
      NAME="$1"
    else
      NAME="$2"
    fi
    
    # default base URL to repositories; change this according to your needs
    BASE="ssh://git@bitbucket.ase.in.tum.de:7999/$EXERCISE/$EXERCISE"
    
    # clone the test repository
    git clone "$BASE-tests.git" "$NAME" && \
      # clone the template repository
      git clone "$BASE-exercise.git" "$NAME/template" && \
      # clone the solution repository
      git clone "$BASE-solution.git" "$NAME/solution" && \
      # create an assignment folder from the template repository
      cp -R "$NAME/template" "$NAME/assignment" && \
      # remove the .git folder from the assignment folder
      rm -r "$NAME/assignment/.git/"
    

• Alternative 2: Open in Artemis (in the browser) and adapt the code in online code editor

  • You can change between the different repos and submit the code when needed

• Alternative 3: Use IntelliJ with the Orion plugin and change the code directly in IntelliJ

  Edit in Editor

  

• Check the results of the template and the solution build plan

• They should not have the status

• In case of a result, some configuration is wrong, please check the build errors on the corresponding build plan.

• Hints: Test cases should only reference code, that is available in the template repository. In case this is not possible, please try out the option Sequential Test Runs

4. Optional: Adapt the build plans

• The build plans are preconfigured and typically do not need to be adapted

• However, if you have additional build steps or different configurations, you can adapt the BASE and SOLUTION build plan as needed

• When students start the programming exercise, the current version of the BASE build plan will be copied. All changes in the configuration will be considered

5. Adapt the interactive problem statement

• Click the button of the programming exercise or navigate into and adapt the interactive problem statement.

• The initial example shows how to integrate tasks, link tests and integrate interactive UML diagrams

6. Configure Grading

7. Verify the exercise configuration

• Open the page of the programming exercise

  

• The template result should have a score of 0% with 0 of X passed

• The solution result should have a score of 100% with X of X passed

• Click on

  • Below the problem statement, you should see Test cases ok and Hints ok

  

Online Editor

The following screenshot shows the online code editor with interactive and dynamic exercise instructions on the right side. Tasks and UML diagram elements are referenced by test cases and update their color from red to green after students submit a new version and all test cases associated with a task or diagram element pass. This allows the students to immediately recognize which tasks are already fulfilled and is particularly helpful for programming beginners.

Online Editor

Testing with Artemis Java Test Sandbox

Artemis Java Test Sandbox (abbr. AJTS) is a JUnit 5 extension for easy and secure Java testing on Artemis.

Its main features are

• a security manager to prevent students crashing the tests or cheating

• more robust tests and builds due to limits on time, threads and io

• support for public and hidden Artemis tests, where hidden ones obey a custom deadline

• utilities for improved feedback in Artemis like processing multiline error messages or pointing to a possible location that caused an Exception

• utilities to test exercises using System.out and System.in comfortably

For more information see https://github.com/ls1intum/artemis-java-test-sandbox