Team Project Computer Graphics and Visualization (SS 2024)

Modules

SWS 0/0/4
INF-B-510, INF-B-520, INF-B-530, INF-B-540,INF-VERT2, INF-VMI-8, INF-VMI-8a, (for INF-VERT7 talk to Prof. Gumhold),INF-MA-PR, INF-E-4, INF-04-KP, MINF-04-KP-FG1, IST-05-KP

SWS 0/0/8
CMS-VC-TEA (Visual Computing Team Project)

Organisation

This year we offer a maximum of three different possible topics as described below. Whether a topic will be offered depends on the number of interested students.

  • We will have our hybrid kickoff meeting on 08.04.2023
  • To register for the course, please enroll in OPAL into one of the topics.
  • Dudle Polls:
    • Human Performance Capture Meeting Timeslot: Dudle Poll
    • Human Performance Capture Subtopic Selection: Dudle Poll
    • Human Performance Capture Role Selection: Dudle Poll

Topic "Spatially Aware 3D Audio Demixing"

Advisor: Mario Henze

The basic premise of this project is to visually recognize sound sources and use this information to dynamically steer a microphone array. One possible scenario would be the recording of a conference, where online-participants can choose to "listen closely" to one or several speakers. To implement this project you will use an Azure Kinect RGBD camera with its builtin microphone array.

The work could be divided into the visual part, which detects sound objects, and the signal part, which uses this spatial information for beam forming. Based on the number of participants and the allocation of kinect cameras, possible extensions would be a multi-camera setup and/or backfeeding acoustic information into the object detection stage.

Requirements

  • Solid programming skills
  • Solid understanding of linear algebra (i.e. coordinate transformations)
  • Beam Forming: Basic knowledge about convolution

Nice-to-have

  • Object Detection: Computer Vision knowledge and/or experience with OpenCV
  • Beam Forming: Experience with programming of DSPs, VSTs, etc.
  • Beam Forming: Experience with SIMD/SIMT-style programming

Topic "Human Performance Capture"

Advisor:  Julien Fischer

The goal of this topic is to create a capturing setup consisting of multiple Azure Kinect RGBD cameras in order to record a human performing various activities, e.g. dancing. The RGB, depth, and audio signals obtained from this recording should be processed by fitting a parameterized human body model to the recordings. To allow the captured human to review their performance, a tool should be developed that allows to replay the recording in AR. The tool should also include common functionality like pausing and unpausing, as well as the possibility to change the location where the performance is played at. Additionally, a function to replace suboptimal parts of the performance with a new capture should be included.

Based on the number of participants, all or only some of the above goals are to be implemented. Since we already have a set of various motion sequences, all topics can be worked on in parallel, such that no hard dependencies exist between the different subtopics.

Requirements

  • Depending on the subtopic, different experiences are of advantage:
  • Recording: solid understanding of (modern) C++ (we can also help with that), basic knowledge of working principles of cameras
  • Processing / Fitting: experience with Machine Learning and Python (optimally PyTorch), basics of computer graphics
  • Replay Tool: computer graphics, first experiences with VR / AR, 

Nice-to-have

  • Recording: experience with multi-camera setups
  • Replay Tool: Knowledge of the framework that you're going to implement the replay tool in

Topic "Sandbox Game"

Advisor: Sebastian Vogt

Picture of the sandbox, below with projection © Bayer, Christian

Based on a student project, we have a sandbox system for you to create a game with. The sandbox is equiped with a Kinnect camera and a projector. Therefor it is possible to estimate the height of the sand at each place in the sandbox and project an image onto the sand. Your task is to create a game using the sandbox as playfield.

The initial idea would be to recreate a terrain in the sand that the computer calculated. The player gets a certain amount of sand to build damms and needs to protect cities from water that is managed by a weather and water simulation. The better/longer the user can protect the cities, the more points he gets.The specifics will be decided during the first group meeting, when we know how many people are interested in the topic. Maybe you also have other interesting ideas that we can discuss.

Basic Tasks

  • conception of a game
  • user interaction and validation with the sand
  • implementation of the game in either Unity or another game engine

Requirements:

  • Motivation
  • Advanced knowledge of Computer Graphics

Nice-to-have

  • Worked with Unity or other game engines

About this page

Stefan Gumhold
Last modified: Apr 09, 2024