Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

My train model using train_celeba.yml #12

Open
fashionguy opened this issue Jul 22, 2020 · 8 comments
Open

My train model using train_celeba.yml #12

fashionguy opened this issue Jul 22, 2020 · 8 comments

Comments

@fashionguy
Copy link

Hi!
I train a model using trian_celeba.yml provided by you without other changes, only change the batch size from 64 to 32.(because my memory has only 8G)
Now I get the trained model after 30 epoches. I run demo.py and get results using my trained model and your model respectively.
But the two results are not exactly the same(eg. the eyes). I don't know why and hope get some advices from you.
(The right is my trained model.)
sendpix0

@dafuny
Copy link

dafuny commented Jul 23, 2020

I have the same question,I trained my model using train_celeba.yml, but I can't get the result as the pretrained model.And then I try to increase the epoch numbers to 50,but the results is still not good.The first one is the pretained model,and the second is my result.Have anyone got some ideas?
图片4
图片3

@dafuny
Copy link

dafuny commented Jul 23, 2020

As I kown ,batch size is related to the learning rate,maybe you can try to decrese your learning rate as the half. @fashionguy

@elliottwu
Copy link
Owner

Hi!
I train a model using trian_celeba.yml provided by you without other changes, only change the batch size from 64 to 32.(because my memory has only 8G)
Now I get the trained model after 30 epoches. I run demo.py and get results using my trained model and your model respectively.
But the two results are not exactly the same(eg. the eyes). I don't know why and hope get some advices from you.
(The right is my trained model.)
sendpix0

Hi!
Thanks for sharing your results. Since 2D-to-3D is a highly ill-posed task and the only objective function here is the reconstruction losses, the model may indeed converge to different feasible solutions. Although your reconstruction results appear to have less details than the pretrained model, the additional details from pretrained model may not be correct, such as the pupils that are sticking out from the eyes.

@elliottwu
Copy link
Owner

elliottwu commented Jul 23, 2020

I have the same question,I trained my model using train_celeba.yml, but I can't get the result as the pretrained model.And then I try to increase the epoch numbers to 50,but the results is still not good.The first one is the pretained model,and the second is my result.Have anyone got some ideas?
图片4
图片3

Hmmm, interesting! This is a scale ambiguity issue (bas-relief ambiguity), but the current parameter setting should converge to a good solution. I have tested it with several random seeds, and it should be rather stable. But let me try to run it again and see if there's any discrepency.

Previously I have found the border depth clamping correlated with the scale. If border depth is clamped too far, it seems to drive the model into predicting a stretched face, like your results. But I can tell this is not case with your results.

@elliottwu
Copy link
Owner

elliottwu commented Jul 28, 2020

Hi, @dafuny! I have double checked the parameters and tested the code on two different machines. The results are consistent with those released. Would like to share your training details in order to investigate the issue?

@dafuny
Copy link

dafuny commented Jul 31, 2020

Hi, @dafuny! I have double checked the parameters and tested the code on two different machines. The results are consistent with those released. Would like to share your training details in order to investigate the issue?

Hi,@elliottwu,it's really kind of you giving me so detailed reply.I have figured out the problem that my conduct environment cuased.Previously,I used cuda10.2,but rencently I installed cuda9.2 and set the same environment as yours.As a results,it can get a good result!The first is your result,and the following is my trained result.
图片2
图片3

@elliottwu
Copy link
Owner

Thanks @dafuny for sharing your results!

It is very interesting that CUDA 10.2 gives completely different convergence! I have never tested it on CUDA 10.2, and was not really aware of it. I might take a closer look into this at some point. Thanks for flagging this!

@hiyijian
Copy link

hiyijian commented Jul 1, 2022

These is no network initialization in this repo. Probably, this is the reason why we get totally diffrient results by using CUDA10.2 and CUDA 9.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants