Assistance System for Dance Creation Using Kinect and Unity

The messages of dance has not been changed until today. Dancing is recognized as the simplest means of communication. When we dance, we do not need words, and dancing is expected to offer opportunities of communication and to improve communication skills(2). a large-scale tournament is held as a sport, Such as baseball and volleyball, and the participants and the audiences from all over the world are impressed. Like this way, dancing also touches people’s hearts. And probably everyone knows how wonderful dancing is. There are 6 million people doing street dance in Japan and street dance has high popularity and status(3).


Introduction (12pt,Bold)
The purpose of this study is to develop a device that can make the choreography of dance easily.
Since ancient times, people have given meanings and various purposes to dance as an action to music such as expressing emotions, praying to God, and transmitting traditional history. The research why people dance (1) leads us to the conclusion : people dance in order to convey what they can not tell with words.
The messages of dance has not been changed until today. Dancing is recognized as the simplest means of communication. When we dance, we do not need words, and dancing is expected to offer opportunities of communication and to improve communication skills(2). a large-scale tournament is held as a sport, Such as baseball and volleyball, and the participants and the audiences from all over the world are impressed. Like this way, dancing also touches people's hearts. And probably everyone knows how wonderful dancing is. There are 6 million people doing street dance in Japan and street dance has high popularity and status(3).
Japanese Ministry of Education announced the martial arts and dance class are compulsory in a junior high school in March 2008 (4),. and they said "Dance is one of the exercises to communicate with friends through exchanging expressions and movements and change image. It is recognized as a movement that we can enjoy feeling emotion with others, capturing images and expressing oneself." This is why Japanese Ministry of Education regard dancing as a required subject.
From these facts, it is obvious that dance is familiar to anyone and it is enjoyed by all ages, genders and tribes all over the world. In order to dance, " Choreographer of the dance" is very important.
The choreographer is the subject of this research. Now, the dance has become more familiar, and dance production will inevitably become more familiar.
It is one of the good examples that the martial arts and dance are compulsory in junior high school described above. The dance classes in the elementary school mainly focus on learning personal skills such as capturing the characteristics of rhythms and dances. But in junior high school, the dance classes mainly focus on group learning and team learning. And it aims at communication and presentation through the process of deepening the expression in dancing. (4). To make a presentation, making choregraph is necessary.
There are another scene that the creative dance is required such as stage appearances at cultural festivals and events at banquets. They are familiar to us.
Although creating dance has become familiar, creating dance is not easy for amateur students and adults. So they cannot create dance right away, and some training is required. I think that there are some solutions to overcome by developing application in choregraphing dance.
The procedures for choreographing dances at entertainments and cultural festivals are mainly described as follows (5).

1.
Define for whom to dance.

2.
Define the place to dance.

3.
Choose a song to use. 4.
Find videos to help you choreograph.

5.
Move in front of the mirror to check. 6.
Write down the choreograph on paper to record it.
Steps 1 to 3 are the foundations for the choreographer. The procedure will determine the direction of choreography.
Step 4 is a search for choreography that fits the direction DOI: 10.12792/iciae2020.012 defined in step 3. It is difficult to design every choreography from the beginning without dance experience. Therefore, it is often used to find videos of experienced dance creators.
Step 5 is to confirm and correct the choreography..
Step 6 is to make a dancing score to explain the choreography to others while making sure to remember the choreography that you created.
The goal of this study is to develop an application or tool that can solve the complains in these procedures and increase the efficiency.
Therefore, in this research, we have decided to take up two points. One of these is in Step 5: "I can't see the movement by myself" and the other is in Step 6 "I forget the choreography I created". The reason we focused on the two points is that dance is difficult to remain as a form comparing other general creation. If the suggestion for dance choreography can be actually confirmed, checking at the mirror will not be necessary, and at the same time there will be no worries about forgetting choreography. Therefore, It can be expected to omit steps 5 and 6.
From these things, the purpose of this research is to make dance formable and simplify the creation of choreography by developing an application that saves and reproduces choreography of dance for students who are not familiar to dance production.
The following applications and tools have already existed. Standing position creation application "daisy beat" that can register the standing position of the dancer in each scene (7).
Step 2 and step 6 can be improved. You can create a dance score which can express a pose with a stickman. Dance score creation application "odorifu" (8).
Step 6 can be improved. A dance video search application "choreograph master dance!" (8).
Step 4 can be improved. However, these apps can only register completed dances. This is not an aid for creating the dance, but for confirming the finished dance. Choreography as an auxiliary system for creating dances. Register the choreography that performed the shooting. By connecting the registered ones, a pattern is created and a series of choreography is created. Succeeded in creating a function that turns a completed thing into one dance. To solve this problem, we have successfully created the basic features of dance creation. Shooting, saving, and playing back choreography with an RGB camera. In addition, we successfully created a gesturebased shooting start function using a kinect sensor so that the RGB camera can automatically perform these shooting operations. By doing so, when the choreography was filmed, it became clear that it was not a choreography information, but simply a record of a "choreography video".
When I recorded the choreography movie, I found it difficult to learn the video because the dancer changed every time and was unnatural, and the distance between the dancer and the recording camera changed every time.
In order to solve this, in this study, we succeeded in overcoming the previous shortcomings by recording the movement of skeletal information and reflecting it in a 3D model.

results of the experiment 2.1 Difference from previous experiment
The purpose of this research is to reproduce the functions of the devices developed so far using 3D models.
Therefore, we decided to use "Unity" as a development environment that allows easy manipulation of 3D models. Save and play back skeleton coordinates detection, connection to 3D model, and choreography as animation. All of these can be reproduced with Unity features. Table 1 below shows the versions of the development environment and the extensions used. Next, the information is embedded in the 3D model, and the 3D model operates. Table 1 Development environment and extension.
In this paper, we propose a system that uses Kinect to read human skeleton information, save the choreography part, and jump to a 3D model of a dance that combines these parts.
By making the 3D model dance using the skeleton information in this way, it is possible to show a smooth dance even if the position, physique, angle, etc. of the person in each part of the  Figure 1 shows skeletal information when Kinect recognizes human movement. Figure 2 shows the person who obtained the skeleton information.

Link between 3D model and skeleton information
The resulting skeletal information is shown in Figure 3. Using this skeleton information, you can move the 3D model as shown in Figure 4. Figure 4 saves figure 5 (3D model movement) as information.

Confirm that there is no difference due to differences in physical information
When taking a choreography record, not all choreography records are captured by the same person.
Regardless of the person who shoots, "recognizing the skeleton information correctly" allows you to continue recording a unified dance.
To test this, you need to experiment with people with different physical information.
Student A (height 172cm, weight 82kg), Student B (height 163cm, weight 54kg) Using the information of these two bodies, experiments using 3D models were performed. Compare the movements of the two students shown in Figure 6 and 7. Action from Figure 8 to Figure 9 and Action from Figure 10 to Figure 11 is the same movement. There was no unnatural behavior. The 3D model did not look unnatural and matched well. This result indicates that only the skeletal information was correctly captured.
This shows that the 3D model behaves the same even with different physical information.

2.4
Make sure there are no differences due to differences in distance In the previous study, trying to connect the choreography as shown in Figure 12 to Figure 13 made it difficult to learn because of the difference in distance from the camera.
In this study, the distance between Figure 14 and Figure 15  This makes it easier to learn than last time.

2.5
Connect choreography to form a dance In this experiment, some dance patterns can be created by performing partial choreography.
By converting choreography to data as shown in Figure 5, you can create a dance by connecting the data as shown by the arrows in Figure 18 and continuously operating the choreography with a 3D model.

Summary of the experiment
In this study, we were able to record skeleton information and link the skeleton information to a 3D model. We succeeded in improving the quality of linking choreography. Also, unlike the previous time, it is a 3D model, so you can move your viewpoint when checking choreography.
As a result, it is now possible to check choreography more closely than before.
Among them, a problem was discovered that if there is a large difference between the skeletal information at the end of choreography and the beginning of choreography, unnatural movements that cannot be called dance occur. To solve this problem, we need to eliminate unnatural movements between choreography. As a result, when choosing the next choreography, you should choose one that has less discomfort when connecting.
To make this happen, we thought we needed the ability to find a good connection and recommend it to the dance creator. We aim to realize this programmatically.