OLD | NEW |
1 **APM Quality Assessment tool** | 1 # APM Quality Assessment tool |
2 | 2 |
3 Python wrapper of audioproc_f with which quality assessment can be automatized. | 3 Python wrapper of `audioproc_f` with which quality assessment can be |
4 The tool allows to simulate different noise conditions, input signals, APM | 4 automatized. The tool allows to simulate different noise conditions, input |
5 configurations and it computes different scores. | 5 signals, APM configurations and it computes different scores. |
6 Once the scores are computed, the results can be easily exported to an HTML page | 6 Once the scores are computed, the results can be easily exported to an HTML page |
7 which allows to listen to the APM input and output signals and also the | 7 which allows to listen to the APM input and output signals and also the |
8 reference one used for evaluation. | 8 reference one used for evaluation. |
9 | 9 |
10 IMPORTANT: **the whole code has not been landed yet.** | 10 ## Dependencies |
| 11 |
| 12 - OS: Linux |
| 13 - Python 2.7 |
| 14 - Python libraries: numpy, scipy, pydub (0.17.0+) |
| 15 - `$ sudo apt install python-numpy python-scipy` |
| 16 - `$ sudo pip install pydub` |
| 17 - PolqaOem64 (see http://www.polqa.info/) |
| 18 - Tested with POLQA Library v1.180 / P863 v2.400 |
| 19 - Aachen Impulse Response (AIR) Database |
| 20 - Download https://www2.iks.rwth-aachen.de/air/air_database_release_1_4.zip |
| 21 - Input probing signals and noise tracks (you can make your own dataset - *1) |
| 22 |
| 23 ## Build |
| 24 |
| 25 - Compile WebRTC |
| 26 - Go to `out/Default/py_quality_assessment` and check that |
| 27 `apm_quality_assessment.py` exists |
| 28 |
| 29 ## First time setup |
| 30 |
| 31 - Deploy PolqaOem64 and set the `POLQA_PATH` environment variable |
| 32 - e.g., `$ export POLQA_PATH=/var/opt/PolqaOem64` |
| 33 - Deploy the AIR Database and set the `AECHEN_IR_DATABASE_PATH` environment |
| 34 variable |
| 35 - e.g., `$ export AECHEN_IR_DATABASE_PATH=/var/opt/AIR_1_4` |
| 36 - Deploy probing signal tracks into |
| 37 - `out/Default/py_quality_assessment/probing_signals` (*1) |
| 38 - Deploy noise tracks into |
| 39 - `out/Default/py_quality_assessment/noise_tracks` (*1, *2) |
| 40 |
| 41 (*1) You can use custom files as long as they are mono tracks sampled at 48kHz |
| 42 encoded in the 16 bit signed format (it is recommended that the tracks are |
| 43 converted and exported with Audacity). |
| 44 |
| 45 (*2) Adapt `EnvironmentalNoiseGenerator._NOISE_TRACKS` accordingly in |
| 46 `out/Default/py_quality_assessment/quality_assessment/noise_generation.py`. |
| 47 |
| 48 ## Usage (scores computation) |
| 49 |
| 50 - Go to `out/Default/py_quality_assessment` |
| 51 - Check the `apm_quality_assessment.sh` as an example script to parallelize the |
| 52 experiments |
| 53 - Adjust the script according to your preferences (e.g., output path) |
| 54 - Run `apm_quality_assessment.sh` |
| 55 - The script will end by opening the browser and showing ALL the computed |
| 56 scores |
| 57 |
| 58 ## Usage (export reports) |
| 59 |
| 60 Showing all the results at once can be confusing. You therefore may want to |
| 61 export separate reports. In this case, you can use the |
| 62 `apm_quality_assessment_export.py` script as follows: |
| 63 |
| 64 - Set --output_dir to the same value used in `apm_quality_assessment.sh` |
| 65 - Use regular expressions to select/filter out scores by |
| 66 - APM configurations: `--config_names, -c` |
| 67 - probing signals: `--input_names, -i` |
| 68 - noise generators: `--noise_generators, -n` |
| 69 - scores: `--eval_scores, -e` |
| 70 - Assign a suffix to the report name using `-f <suffix>` |
| 71 |
| 72 For instance: |
| 73 |
| 74 ``` |
| 75 $ ./apm_quality_assessment-export.py \ |
| 76 -o ~/data/apm_quality_assessment \ |
| 77 -e \(polqa\) \ |
| 78 -n \(echo\) \ |
| 79 -c "(^default$)|(.*AE.*)" \ |
| 80 -f echo |
| 81 ``` |
| 82 |
| 83 ## Troubleshooting |
| 84 |
| 85 The input wav file must be: |
| 86 - sampled at a sample rate that is a multiple of 100 (required by POLQA) |
| 87 - in the 16 bit format (required by `audioproc_f`) |
| 88 - encoded in the Microsoft WAV signed 16 bit PCM format (Audacity default |
| 89 when exporting) |
| 90 |
| 91 Depending on the license, the POLQA tool may take “breaks” as a way to limit the |
| 92 throughput. When this happens, the APM Quality Assessment tool is slowed down. |
| 93 For more details about this limitation, check Section 10.9.1 in the POLQA manual |
| 94 v.1.18. |
| 95 |
| 96 In case of issues with the POLQA score computation, check |
| 97 `py_quality_assessment/eval_scores.py` and adapt |
| 98 `PolqaScore._parse_output_file()`. |
| 99 The code can be also fixed directly into the build directory (namely, |
| 100 `out/Default/py_quality_assessment/eval_scores.py`). |
OLD | NEW |