devAIce®

The VAD module detects the presence of voice in an audio stream. The detection is strongly robust to noise and independent of the volume level of the voice. This means that also faint voices can be detected in the presence of louder background noises.

Detecting voice in large amounts of audio material leads to a resource-saving and efficient analysis process. If VAD runs before the voice analysis itself, large amounts of non-voice data can be filtered and excluded from the analysis.

The Emotion module performs emotion recognition on voice. The module is designed to detect emotions in all languages. Currently, the module combines two independent emotion models with different output representations:

• A dimensional arousal-valence-dominance emotion model
• A four-class categorical emotion model: happy, angry, sad, and neutral

In devAIce® we offer two Emotion modules: Emotion & Emotion Large. 

If working with limited computational resources or running your application on devices with lower memory, the Emotion module is a better fit for your needs, compared to the Emotion Large module, which has a higher prediction accuracy.

The Multi-Modal Emotion module combines acoustics- and linguistics-based emotion recognition in a single module. It achieves higher accuracy than models that are limited to only one of the modalities (e.g. the model provided by the Emotion module). Acoustic models tend to perform better at estimating the arousal dimension of emotions, while linguistic models excel at predicting valence (positivity/negativity). The Multi-Modal Emotion module fuses information from both modalities to improve the prediction accuracy.

The Acoustic Scene Module distinguishes between 3 classes:

• Indoor 
• Outdoor
• Transport

Further subclasses are recognized in each acoustic scene class. This model is currently under development – the specific subclasses will be named in the next update.

The AED module runs acoustic event detection for multiple acoustic event categories on an audio stream.

Currently, speech and music are supported acoustic event categories. The model allows events of different categories to overlap temporally.

The Speaker Attributes module estimates the personal attributes of speakers from voice and speech. These attributes include:

• A perceived gender (sex), divided into the categories:
  • female adult
  • male adult
  • child
• A perceived age, in years

devAIce® provides two gender submodules called Gender and Gender (Small).

The age-gender models are trained on self-reported gender.

Evaluate whether the speaker in a recording is the same as a previously-enrolled reference speaker.
Two modes are supported:

• Enrollment mode: a speaker model for a reference speaker is created or updated based on one or more reference recordings.
• Verification mode: a previously created speaker model is used to estimate how likely the same speaker is present in a given recording.
  
  

*Speaker Verification is currently in development and available in Beta version.

The Prosody module computes the following prosodic features:

• F0 (in Hz)
• Loudness
• Speaking rate (in syllables per second)
• Intonation

Running Prosody module in combination with VAD:

• If the VAD module is disabled, the full audio input is analyzed as a single utterance, and one prosody result is generated.
• When VAD is enabled, the audio input is first segmented by the VAD before the Prosody module is run on each detected voice segment. In this case, individual prosody results for each segment will be output.

The openSMILE Features module performs feature extraction on speech. Currently, it includes the following two feature sets based on openSMILE:

• ComParE-2016

This feature set consists of a total of 6373 audio features that are constructed by extracting energy, voicing, and spectral low-level descriptors and computing statistical functionals on them such as percentiles, moments, peaks, and temporal features.

While originally designed for the task of speaker emotion recognition, it has been shown to also work well for a wide range of other audio classification tasks.

• GeMAPS+

The GeMAPS+ feature set is a proprietary extension of the GeMAPS feature set described in Eyben et al.2. The set consists of a total of 276 audio features and has been designed as a minimalistic, general-purpose set for common analysis and classification tasks on voice.