Reproducing parallel results

[mortont]

I've been trying to reproduce the parallel wavenet results, however I'm running into some issues with training the teacher model. I have trained it on the LJ Speech dataset with the default wavenet_mol.json configuration to ~280k steps (all other hparams are unchanged as well). The loss looks good, however the evaluated speech is just babbling, as if local conditioning wasn't used.

I didn't see anything immediately apparent as to why this is happening, do you have any ideas?

[bfs18]

Hi @mortont
Could you provide more information? For example, the loss curve, some generated samples and the code commit.

[mortont]

Sure, I'm on commit 67eacb995aef465d1e2ed810f25e0d7d3899e9b6 and this is the loss curve

This is one of the generated samples: gen_LJ001-0028.wav.zip
For reference, this is the original that was used for the mel conditioning: LJ001-0028.wav.zip

What loss did you let the teacher get to in your examples? Wondering if this just needs more training time. Let me know if any other information would be helpful.

[bfs18]

According to my experience, if a small batch size such as 2 is used, 200k training steps is not enough. Because the model haven't seen enough data. What's your batch size? Besides please ensure USE_RESIZE_CONV=False in masked.py.
This is my loss curve.

[mortont]

Ah, thank you. My batch size is 2 so I'll continue training. I'll go ahead and close this issue and report back when I have good samples for posterity.

[bfs18]

Hi @mortont , I'm sorry that the wavenet results cannot be reproduced with the default config json. Recently I cannot reporduce the previous result either. And I finally figure out that weight normalization harms the wavenet performance. So setting use_weigh_norm=false in wavenet_mol.json may solve your problem.
Initially weight normalization produces some promising results for parallel wavenet. So I keep it as a default configuration. Probabily it has negative effects on the learned mel-sepctrum representation. And good mel-spectrum representation is vital for a good modle, since the wave sample points totaly depend on theit.
I'm also doing simaliar tests on parallel wavenet.
Once again, I'm sorry.

[mortont]

Good catch @bfs18, thank you! I'll try training with use_weight_norm as false. Unfortunately my training is a bit slow with a single GPU. Do you have a pretrained teacher model to share? I'd be happy to help on any parts of parallel wavenet once I have a good teacher.

[mortont]

Actually, I just saw the updated readme... I'll check that out now.

[bfs18]

Hi,
I tested the code, setting use_weigth_norm=False solves the problem.
And I implemented http://export.arxiv.org/abs/1807.07281 at this weekend. I will update the code after some testing.

[zhang-jian]

I am getting error when loading the pre-trained model https://drive.google.com/open?id=13rHT6zr2sXeedmjUOpp6IVQdT30cy66_

  File "/scratch/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tensorflow-1.6.0-wm4rkg4qsenbhfdz7fzlopz6qcxckfff/lib/python3.4/site-packages/tensorflow/python/training/saver.py", line 1812, in latest_checkpoint
    if file_io.get_matching_files(v2_path) or file_io.get_matching_files(
  File "/scratch/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tensorflow-1.6.0-wm4rkg4qsenbhfdz7fzlopz6qcxckfff/lib/python3.4/site-packages/tensorflow/python/lib/io/file_io.py", line 337, in get_matching_files
    for single_filename in filename
  File "/scratch/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tensorflow-1.6.0-wm4rkg4qsenbhfdz7fzlopz6qcxckfff/lib/python3.4/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /data/logs-wavenet/eval_ckpt/ns_pwn-n_MU-WN_DDI_mfinit-n_LOGS-n_CLIP-MAG-L2-06_27; No such file or directory

Any idea?

[bfs18]

Hi @zhang-jian, I'm sorry for that.
The first line of ns_pwn-eval/checkpoint contains the absolute path of the model checkpoint. You can modify the path according to your file system. Or you can only keep the basename of that path. Refer to ns_wn-eval/checkpoint for an example.

[mortont]

I was able to confirm that the pretrained model produces a recognizable voice, all that was needed was changing the path in the checkpoint file to a relative one. Great work!

@bfs18 ClariNet looks very interesting, mostly because of the more numerically stable implementation. I've noticed that parallel wavenet optimization is very difficult and unstable, so hopefully ClariNet helps with that.

[bfs18]

I got some new examples running with contrastive loss and without weight normalization at step 70948. The result may improve a bit after longer running.
gen_LJ001-0001
gen_LJ001-0001

[bfs18]

I look deeper into the problem.
Some bad configuration (e.g. weight normalization + tanh trans conv act) may cause the activation of the transposed convolution layer saturate. So the mel condition becomes meaningless. The model degenerate to an unconditional one.
The following figures are the histogram and spectrum of the transposed convolution stack output. This model only generate random speech even though mel condition is used.


In contrast, the following figures come from an OK model.


Most of the activation values are close to 0. So the learned representation may be considered sparse.
I think there are 2 solutions.

1. use activation functions that would not saturate, e.g. leaky_relu in Clarinet.
2. When teacher forcing is used, the model can predict the original waveform totally conditioned on the teacher forcing input. The teacher forcing input contains the complete information. So we can use dropout to make the teacher forcing input incomplete. Then the model is forced to access the additional mel condition data to predict the original waveform. I'm not sure to which layers dropout should be applied. I am working on this.
3. add noise to teacher forcing inputs. The previous adding noise implementation is buggy. Because I added noise both to inputs and outputs. So the predicted wave is noisy. I will fix this.