External denoiser based on noise estimation and moving object detection.

webrtc/modules/video_processing/util/noise_estimation.cc

Index: webrtc/modules/video_processing/util/noise_estimation.cc
diff --git a/webrtc/modules/video_processing/util/noise_estimation.cc b/webrtc/modules/video_processing/util/noise_estimation.cc
new file mode 100644
index 0000000000000000000000000000000000000000..3367fee27305967c61ca8071ccb1e64fa18c2dee
--- /dev/null
+++ b/webrtc/modules/video_processing/util/noise_estimation.cc
@@ -0,0 +1,85 @@
+/*
+ *  Copyright (c) 2016 The WebRTC project authors. All Rights Reserved.
+ *
+ *  Use of this source code is governed by a BSD-style license
+ *  that can be found in the LICENSE file in the root of the source
+ *  tree. An additional intellectual property rights grant can be found
+ *  in the file PATENTS.  All contributing project authors may
+ *  be found in the AUTHORS file in the root of the source tree.
+ */
+
+#include "webrtc/modules/video_processing/util/noise_estimation.h"
+
+namespace webrtc {
+
+void NoiseEstimation::Init(int width, int height, CpuType cpu_type) {
+  int mb_cols = width >> 4;
+  int mb_rows = height >> 4;
+  consec_low_var_.reset(new uint32_t[mb_cols * mb_rows]());
+  width_ = width;
+  height_ = height;
+  cpu_type_ = cpu_type;
+}
+
+void NoiseEstimation::GetNoise(int mb_index, uint32_t var, uint32_t luma) {

[marpan, 2016/03/30 18:27:20]

Comments: This computes block noise and accumulates?

[jackychen_, 2016/03/30 22:36:37]

It normalizes var and accumulates. var is computed outside, I actually want to
move it in to save some cycles (if we do noise estimation less frequent, we save
one variance computation). Since it is related to speed-up, I postpone it to
next cl.

+  consec_low_var_[mb_index]++;
+  num_stationary_block_++;
+  if (consec_low_var_[mb_index] >= kConsecLowVarFrame &&
+      (luma >> 8) < kAverageLumaMax && (luma >> 8) > kAverageLumaMin) {
+    int nor_var = var / (luma >> 12);
+    noise_var_ +=
+        nor_var > kBlockSelectionVarMax ? kBlockSelectionVarMax : nor_var;
+    num_noisy_block_++;
+  }
+}
+
+void NoiseEstimation::ResetConsecLowVar(int mb_index) {
+  consec_low_var_[mb_index] = 0;
+}
+
+uint8_t NoiseEstimation::GetNoiseLevel() {

[marpan, 2016/03/30 18:27:20]

You're still doing 3 things in this function:
1. normalize the frame-level noise estimate
2. update with recursive average
3. and threshold to get noise level

Still feel splitting this into at least 2 functions would be better.

Also some comments to what function(s) does.

[jackychen_, 2016/03/30 22:36:37]

Split to 2 functions and add some comments.

+  int mb_cols = width_ >> 4;
+  int mb_rows = height_ >> 4;

[marpan, 2016/03/30 18:27:20]

is it worth defining mb_cols/rows_ in this class, so only need to set once in
Init.

[jackychen_, 2016/03/30 22:36:37]

Done.

+  // No enough samples implies the motion of the camera or too many moving

[marpan, 2016/03/30 18:27:20]

Should condition of "Not enough samples" be num_noisy_block > thresh ?

[jackychen_, 2016/03/30 22:36:37]

Correct. I just use thresh = 0, so I code as !num_noisy_block. It's actually a
weak condition, but works well with the first part of the condition.

+  // objects in the frame.
+  if (num_stationary_block_ < 0.65 * mb_cols * mb_rows || !num_noisy_block_) {

[marpan, 2016/03/30 18:27:20]

prefer  (0.65 * mb_cols * mb_rows)

[jackychen_, 2016/03/30 22:36:37]

Done.

+    noise_var_ = 0;
+    noise_var_accum_ = 0;

[marpan, 2016/03/30 18:27:20]

is this intentional to set (the running average) noise_var_accum_ = 0 if a frame
suddenly has alot of motion?

[jackychen_, 2016/03/30 22:36:37]

Yes. When it happens, I think we should start over. The logic may change if we
compute noise level less frequent, say every 10 frames.

+    num_stationary_block_ = 0;
+    num_noisy_block_ = 0;

[marpan, 2016/03/30 18:27:20]

the 3 quantities (noise_var, num_stationary_block_, num_noisy_block_) seem to
get reset
anyway below. Maybe can we do it only once at end of function?

[jackychen_, 2016/03/30 22:36:37]

There is a "return" in the last line of if branch, not easy to find. So need to
keep both.

+#if DISPLAY
+    printf("Not enough samples.\n");
+#endif
+    return 0;
+  } else {
+    noise_var_ /= num_noisy_block_;
+    percent_non_montion_block_ =

[marpan, 2016/03/30 18:27:20]

non_motion

[jackychen_, 2016/03/30 22:36:37]

Maybe it's better to use "static" to make it concise. 

+        static_cast<double>(num_stationary_block_) / (mb_cols * mb_rows);
+#if DISPLAY
+    printf("%d %d fraction = %.3f\n", num_stationary_block_, mb_cols * mb_rows,
+           percent_non_montion_block_);
+#endif
+    num_noisy_block_ = 0;
+    num_stationary_block_ = 0;
+  }
+  // For the first frame get the noise value, just use the value as accumulated
+  // one, otherwise, use the averaging window.
+  if (noise_var_accum_ == 0) {
+    noise_var_accum_ = noise_var_;
+  } else {
+    noise_var_accum_ = (noise_var_accum_ * 15 + noise_var_) / 16;
+  }
+#if DISPLAY
+  printf("noise_var_accum_ = %.1f, noise_var_ = %d.\n", noise_var_accum_,
+         noise_var_);
+#endif
+  // Reset noise_var_ for the next frame.
+  noise_var_ = 0;
+  int noise_thr = cpu_type_ ? kNoiseThreshold : kNoiseThresholdNeon;
+  if (noise_var_accum_ > noise_thr) {
+    return 1;
+  }
+  return 0;
+}
+
+}  // namespace webrtc