Use VAD to get a better speech power estimation in the IntelligibilityEnhancer

webrtc/modules/audio_processing/intelligibility/intelligibility_utils.h

 /*
  *  Copyright (c) 2014 The WebRTC project authors. All Rights Reserved.
  *
  *  Use of this source code is governed by a BSD-style license
  *  that can be found in the LICENSE file in the root of the source
  *  tree. An additional intellectual property rights grant can be found
  *  in the file PATENTS.  All contributing project authors may
  *  be found in the AUTHORS file in the root of the source tree.
  */
 
-//
-//  Specifies helper classes for intelligibility enhancement.
-//
-
 #ifndef WEBRTC_MODULES_AUDIO_PROCESSING_INTELLIGIBILITY_INTELLIGIBILITY_UTILS_H_
 #define WEBRTC_MODULES_AUDIO_PROCESSING_INTELLIGIBILITY_INTELLIGIBILITY_UTILS_H_
 
 #include <complex>
+#include <vector>
 
 #include "webrtc/base/scoped_ptr.h"
 
 namespace webrtc {
 
 namespace intelligibility {
 
-// Return |current| changed towards |target|, with the change being at most
-// |limit|.
-float UpdateFactor(float target, float current, float limit);
+// Internal helper for computing the power of a stream of arrays.
+// The result is an array of power per position: the i-th power is the power of
+// the stream of data on the i-th positions in the input arrays.
+class PowerEstimator {
+ public:
+  // Construct an instance for the given input array length (|freqs|), with the
+  // appropriate parameters. |decay| is the forgetting factor.
+  PowerEstimator(size_t freqs, float decay);
 
-// Apply a small fudge to degenerate complex values. The numbers in the array
-// were chosen randomly, so that even a series of all zeroes has some small
-// variability.
-std::complex<float> zerofudge(std::complex<float> c);
+  // Add a new data point to the series.
+  template <typename T>

[turaj, 2016/02/19 16:48:47]

I'm sure have compiled this code, just out of my curiosity, usually templates
need instantiations, wasn't that necessary here? And why implementation is in
header file, while implementation of constructor is in .cc file?

I'm not experienced with templates, but I would have templated the class and
instantiate the possible templates. I guess in this case instantiations would be
<float> and <std::complex<float>>. 

[aluebs-webrtc, 2016/02/19 19:30:48]

I am not experienced with templates either, but I decided to only templatize the
function, since the implementation is the same and power_ has the same type,
changing only the input type of that function. But maybe it is preferred to
templatize the whole class, I am not sure.
Templates only need explicit instantiations when implemented in the cc file
instead of the header, so I implemented the template function in the header to
get the additional flexibility of not having to explicitly instantiate.
If you prefer a class template, I am happy to change it, since I am not sure
myself what the criteria is to chose between function and class template. Or if
you prefer to have the implementation in the cc file and restrict the template
with explicit instantations.

[turaj, 2016/02/22 17:36:53]

Like you, I really don't know what is preferred with respect to templating
member function vs templating class, nor header-file implementation versus
source-file implementation. However, the reason I proposed to have template,
beside saving couple of lines of code, was to prevent a potential bug my mixing
complex and float averaging. If that concern is valid (which I'm not sure as
this is an internal class and only intelligibility is using it) then templating
like this is not serving the purpose. Things can still go wrong. 

Templating class, on the other hand, prevents the above issue as intelligibility
will have

PowerEstimator<float> noise_power_;
PowerEstimator<std::complex<float>> speech_power_;

Given the above, I a bit prefer class templating, but I don't have strong
opinion, so I leave it up to you.

[aluebs-webrtc, 2016/02/22 20:41:03]

That is an excellent point! Changed it to a template class.

+  void Step(const T* data) {
+    for (size_t i = 0; i < power_.size(); ++i) {
+      power_[i] = decay_ * power_[i] +
+                  (1.f - decay_) * std::abs(data[i]) * std::abs(data[i]);
+    }
+  }
 
-// Incremental mean computation. Return the mean of the series with the
-// mean |mean| with added |data|.
-std::complex<float> NewMean(std::complex<float> mean,
-                            std::complex<float> data,
-                            size_t count);
-
-// Updates |mean| with added |data|;
-void AddToMean(std::complex<float> data,
-               size_t count,
-               std::complex<float>* mean);
-
-// Internal helper for computing the variances of a stream of arrays.
-// The result is an array of variances per position: the i-th variance
-// is the variance of the stream of data on the i-th positions in the
-// input arrays.
-// There are four methods of computation:
-//  * kStepInfinite computes variances from the beginning onwards
-//  * kStepDecaying uses a recursive exponential decay formula with a
-//    settable forgetting factor
-//  * kStepWindowed computes variances within a moving window
-//  * kStepBlocked is similar to kStepWindowed, but history is kept
-//    as a rolling window of blocks: multiple input elements are used for
-//    one block and the history then consists of the variances of these blocks
-//    with the same effect as kStepWindowed, but less storage, so the window
-//    can be longer
-class VarianceArray {
- public:
-  enum StepType {
-    kStepInfinite = 0,
-    kStepDecaying,
-    kStepWindowed,
-    kStepBlocked,
-    kStepBlockBasedMovingAverage
-  };
-
-  // Construct an instance for the given input array length (|freqs|) and
-  // computation algorithm (|type|), with the appropriate parameters.
-  // |window_size| is the number of samples for kStepWindowed and
-  // the number of blocks for kStepBlocked. |decay| is the forgetting factor
-  // for kStepDecaying.
-  VarianceArray(size_t freqs, StepType type, size_t window_size, float decay);
-
-  // Add a new data point to the series and compute the new variances.
-  // TODO(bercic) |skip_fudge| is a flag for kStepWindowed and kStepDecaying,
-  // whether they should skip adding some small dummy values to the input
-  // to prevent problems with all-zero inputs. Can probably be removed.
-  void Step(const std::complex<float>* data, bool skip_fudge = false) {
-    (this->*step_func_)(data, skip_fudge);
-  }
-  // Reset variances to zero and forget all history.
-  void Clear();
-  // Scale the input data by |scale|. Effectively multiply variances
-  // by |scale^2|.
-  void ApplyScale(float scale);
-
-  // The current set of variances.
-  const float* variance() const { return variance_.get(); }
-
-  // The mean value of the current set of variances.
-  float array_mean() const { return array_mean_; }
+  // The current power array.
+  const std::vector<float>& power() { return power_; };
 
  private:
-  void InfiniteStep(const std::complex<float>* data, bool dummy);
-  void DecayStep(const std::complex<float>* data, bool dummy);
-  void WindowedStep(const std::complex<float>* data, bool dummy);
-  void BlockedStep(const std::complex<float>* data, bool dummy);
-  void BlockBasedMovingAverage(const std::complex<float>* data, bool dummy);
+  // The current power array.
+  std::vector<float> power_;
 
-  // TODO(ekmeyerson): Switch the following running means
-  // and histories from rtc::scoped_ptr to std::vector.
-
-  // The current average X and X^2.
-  rtc::scoped_ptr<std::complex<float>[]> running_mean_;
-  rtc::scoped_ptr<std::complex<float>[]> running_mean_sq_;
-
-  // Average X and X^2 for the current block in kStepBlocked.
-  rtc::scoped_ptr<std::complex<float>[]> sub_running_mean_;
-  rtc::scoped_ptr<std::complex<float>[]> sub_running_mean_sq_;
-
-  // Sample history for the rolling window in kStepWindowed and block-wise
-  // histories for kStepBlocked.
-  rtc::scoped_ptr<rtc::scoped_ptr<std::complex<float>[]>[]> history_;
-  rtc::scoped_ptr<rtc::scoped_ptr<std::complex<float>[]>[]> subhistory_;
-  rtc::scoped_ptr<rtc::scoped_ptr<std::complex<float>[]>[]> subhistory_sq_;
-
-  // The current set of variances and sums for Welford's algorithm.
-  rtc::scoped_ptr<float[]> variance_;
-  rtc::scoped_ptr<float[]> conj_sum_;
-
-  const size_t num_freqs_;
-  const size_t window_size_;
   const float decay_;
-  size_t history_cursor_;
-  size_t count_;
-  float array_mean_;
-  bool buffer_full_;
-  void (VarianceArray::*step_func_)(const std::complex<float>*, bool);
 };
 
-// Helper class for smoothing gain changes. On each applicatiion step, the
+// Helper class for smoothing gain changes. On each application step, the
 // currently used gains are changed towards a set of settable target gains,
-// constrained by a limit on the magnitude of the changes.
+// constrained by a limit on the relative changes.
 class GainApplier {
  public:
-  GainApplier(size_t freqs, float change_limit);
+  GainApplier(size_t freqs, float relative_change_limit);
 
   // Copy |in_block| to |out_block|, multiplied by the current set of gains,
   // and step the current set of gains towards the target set.
   void Apply(const std::complex<float>* in_block,
              std::complex<float>* out_block);
 
   // Return the current target gain set. Modify this array to set the targets.
   float* target() const { return target_.get(); }
 
  private:
   const size_t num_freqs_;
-  const float change_limit_;
+  const float relative_change_limit_;
   rtc::scoped_ptr<float[]> target_;
   rtc::scoped_ptr<float[]> current_;
 };
 
 }  // namespace intelligibility
 
 }  // namespace webrtc
 
 #endif  // WEBRTC_MODULES_AUDIO_PROCESSING_INTELLIGIBILITY_INTELLIGIBILITY_UTILS_H_