OLD | NEW |
1 # APM Quality Assessment tool | 1 # APM Quality Assessment tool |
2 | 2 |
3 Python wrapper of `audioproc_f` with which quality assessment can be | 3 Python wrapper of `audioproc_f` with which quality assessment can be |
4 automatized. The tool allows to simulate different noise conditions, input | 4 automatized. The tool allows to simulate different noise conditions, input |
5 signals, APM configurations and it computes different scores. | 5 signals, APM configurations and it computes different scores. |
6 Once the scores are computed, the results can be easily exported to an HTML page | 6 Once the scores are computed, the results can be easily exported to an HTML page |
7 which allows to listen to the APM input and output signals and also the | 7 which allows to listen to the APM input and output signals and also the |
8 reference one used for evaluation. | 8 reference one used for evaluation. |
9 | 9 |
10 ## Dependencies | 10 ## Dependencies |
11 | 11 |
12 - OS: Linux | 12 - OS: Linux |
13 - Python 2.7 | 13 - Python 2.7 |
14 - Python libraries: numpy, scipy, pydub (0.17.0+) | 14 - Python libraries: numpy, scipy, pydub (0.17.0+), pandas (0.20.1+) |
15 - It is recommended that a dedicated Python environment is used | 15 - It is recommended that a dedicated Python environment is used |
16 - install `virtualenv` | 16 - install `virtualenv` |
17 - `$ sudo apt-get install python-virtualenv` | 17 - `$ sudo apt-get install python-virtualenv` |
18 - setup a new Python environment (e.g., `my_env`) | 18 - setup a new Python environment (e.g., `my_env`) |
19 - `$ cd ~ && virtualenv my_env` | 19 - `$ cd ~ && virtualenv my_env` |
20 - activate the new Python environment | 20 - activate the new Python environment |
21 - `$ source ~/my_env/bin/activate` | 21 - `$ source ~/my_env/bin/activate` |
22 - add dependcies via `pip` | 22 - add dependcies via `pip` |
23 - `(my_env)$ pip install numpy pydub scipy` | 23 - `(my_env)$ pip install numpy pydub scipy pandas` |
24 - PolqaOem64 (see http://www.polqa.info/) | 24 - PolqaOem64 (see http://www.polqa.info/) |
25 - Tested with POLQA Library v1.180 / P863 v2.400 | 25 - Tested with POLQA Library v1.180 / P863 v2.400 |
26 - Aachen Impulse Response (AIR) Database | 26 - Aachen Impulse Response (AIR) Database |
27 - Download https://www2.iks.rwth-aachen.de/air/air_database_release_1_4.zip | 27 - Download https://www2.iks.rwth-aachen.de/air/air_database_release_1_4.zip |
28 - Input probing signals and noise tracks (you can make your own dataset - *1) | 28 - Input probing signals and noise tracks (you can make your own dataset - *1) |
29 | 29 |
30 ## Build | 30 ## Build |
31 | 31 |
32 - Compile WebRTC | 32 - Compile WebRTC |
33 - Go to `out/Default/py_quality_assessment` and check that | 33 - Go to `out/Default/py_quality_assessment` and check that |
(...skipping 27 matching lines...) Expand all Loading... |
61 - Run `apm_quality_assessment.sh` | 61 - Run `apm_quality_assessment.sh` |
62 - The script will end by opening the browser and showing ALL the computed | 62 - The script will end by opening the browser and showing ALL the computed |
63 scores | 63 scores |
64 | 64 |
65 ## Usage (export reports) | 65 ## Usage (export reports) |
66 | 66 |
67 Showing all the results at once can be confusing. You therefore may want to | 67 Showing all the results at once can be confusing. You therefore may want to |
68 export separate reports. In this case, you can use the | 68 export separate reports. In this case, you can use the |
69 `apm_quality_assessment_export.py` script as follows: | 69 `apm_quality_assessment_export.py` script as follows: |
70 | 70 |
71 - Set --output_dir to the same value used in `apm_quality_assessment.sh` | 71 - Set `--output_dir, -o` to the same value used in `apm_quality_assessment.sh` |
72 - Use regular expressions to select/filter out scores by | 72 - Use regular expressions to select/filter out scores by |
73 - APM configurations: `--config_names, -c` | 73 - APM configurations: `--config_names, -c` |
74 - probing signals: `--input_names, -i` | 74 - capture signals: `--capture_names, -i` |
| 75 - render signals: `--render_names, -r` |
| 76 - echo simulator: `--echo_simulator_names, -e` |
75 - test data generators: `--test_data_generators, -t` | 77 - test data generators: `--test_data_generators, -t` |
76 - scores: `--eval_scores, -e` | 78 - scores: `--eval_scores, -s` |
77 - Assign a suffix to the report name using `-f <suffix>` | 79 - Assign a suffix to the report name using `-f <suffix>` |
78 | 80 |
79 For instance: | 81 For instance: |
80 | 82 |
81 ``` | 83 ``` |
82 $ ./apm_quality_assessment-export.py \ | 84 $ ./apm_quality_assessment-export.py \ |
83 -o ~/data/apm_quality_assessment \ | 85 -o output/ \ |
84 -e \(polqa\) \ | |
85 -n \(echo\) \ | |
86 -c "(^default$)|(.*AE.*)" \ | 86 -c "(^default$)|(.*AE.*)" \ |
| 87 -t \(white_noise\) \ |
| 88 -s \(polqa\) \ |
87 -f echo | 89 -f echo |
88 ``` | 90 ``` |
89 | 91 |
90 ## Troubleshooting | 92 ## Troubleshooting |
91 | 93 |
92 The input wav file must be: | 94 The input wav file must be: |
93 - sampled at a sample rate that is a multiple of 100 (required by POLQA) | 95 - sampled at a sample rate that is a multiple of 100 (required by POLQA) |
94 - in the 16 bit format (required by `audioproc_f`) | 96 - in the 16 bit format (required by `audioproc_f`) |
95 - encoded in the Microsoft WAV signed 16 bit PCM format (Audacity default | 97 - encoded in the Microsoft WAV signed 16 bit PCM format (Audacity default |
96 when exporting) | 98 when exporting) |
97 | 99 |
98 Depending on the license, the POLQA tool may take “breaks” as a way to limit the | 100 Depending on the license, the POLQA tool may take “breaks” as a way to limit the |
99 throughput. When this happens, the APM Quality Assessment tool is slowed down. | 101 throughput. When this happens, the APM Quality Assessment tool is slowed down. |
100 For more details about this limitation, check Section 10.9.1 in the POLQA manual | 102 For more details about this limitation, check Section 10.9.1 in the POLQA manual |
101 v.1.18. | 103 v.1.18. |
102 | 104 |
103 In case of issues with the POLQA score computation, check | 105 In case of issues with the POLQA score computation, check |
104 `py_quality_assessment/eval_scores.py` and adapt | 106 `py_quality_assessment/eval_scores.py` and adapt |
105 `PolqaScore._parse_output_file()`. | 107 `PolqaScore._parse_output_file()`. |
106 The code can be also fixed directly into the build directory (namely, | 108 The code can be also fixed directly into the build directory (namely, |
107 `out/Default/py_quality_assessment/eval_scores.py`). | 109 `out/Default/py_quality_assessment/eval_scores.py`). |
OLD | NEW |