can't reproduce results even set all random seeds

[zzd1992]

My environment： python2.7, cuda8.0, cudnn, pytorch 0.3.1
I set all random seeds but I still can't reproduce results. Here is part of my code:

torch.manual_seed(0)
torch.cuda.manual_seed(0)
np.random.seed(0)

transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (1.0, 1.0, 1.0)),
    #transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (1.0, 1.0, 1.0)),
    #transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])


trainset = torchvision.datasets.CIFAR10(root='./data', train=True, transform=transform_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=args.batchsize, shuffle=True, num_workers=0)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, transform=transform_test)
testloader = torch.utils.data.DataLoader(testset, batch_size=200, shuffle=False, num_workers=0)

network = model.Cifar()
network.cuda()

I am sure there is no randomness introduced by extra opreators.
I guess the problem is caused by transfroms or DataLoader.
Any one can help me ?

[fmassa]

I believe you also need to set random.seed(0), as it's used by some of the random transforms.
Also, the threads in the DataLoader will have different seeds prior to v0.4, so you should probably want to update to PyTorch 0.4 (which fixes the seeds of the DataLoader threads).
Finally, there is non-determinism in some cudnn functions. If you really care about determinism, try setting torch.backends.cudnn.deterministic=True.

Closing as not a bug in PyTorch. Feel free to reopen if you think this is actually a bug.

[aabobakr]

I tried all the following but the results remained stochastic:

torch.manual_seed(12)
torch.cuda.manual_seed(12)
np.random.seed(12)
random.seed(12)

torch.backends.cudnn.deterministic=True

The only thing that worked was also setting num_workers=0 in the DataLoader constructor. I am using PyTorch 0.5 built from source.

[fmassa]

You probably also need to pass the np.random.seed(12) in the worker_init_fn to make it deterministic. Then you should probably be able to use multiple workers.

[aabobakr]

Thanks @fmassa, I tried that as well but didn't work

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True,
                                           num_workers=args.workers, pin_memory=True, worker_init_fn=np.random.seed(12))

val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=args.batch_size, shuffle=False,
                                         num_workers=args.workers, pin_memory=True, worker_init_fn=np.random.seed(12))

[zzd1992]

@aabobakr I try “torch.backends.cudnn.deterministic=True” in torch v0.4, and it works.
Maybe you need also pass random seed for "train_loader" or maybe torch v0.5 is different from v0.4

[aabobakr]

@zzd1992 I passed the random seed to both the train and val loader, "updated my comment". Maybe there is a difference in 0.5, I think I will stick to 0 workers for now.

[fmassa]

@aabobakr I don't know how your code didn't crash with the init function that you passed.
It should be instead something like

def _init_fn(worker_id):
    np.random.seed(12 + worker_id)

or you can completely ignore the worker_id if you want.

[aabobakr]

@fmassa I was expecting it to crash too as np.random.seed(12) is not init function at all, but it didn't. But thank you it works now.

[isalirezag]

@fmassa it does sound like that it is not working for pytorch 1.0.1
I check it and still get random results. can you please check it ?

[vijayendra-g]

@fmassa
torch.manual_seed(42)
torch.cuda.manual_seed(42)
np.random.seed(42)
random.seed(42)
torch.backends.cudnn.deterministic=True

features = torch.randn(2,5)

I get different results in features everytime.

I am using pytorch 1.0.0 in jupyter notebook

[sbelharbi]

Actually, I am facing some unusual non-determinism in Pytorch 1.0.0. I did not investigate that yet. But your comment intrigued me. (Pytorch note on randomness). However, your code does seem to give the same random numbers on my computer (CPU), if I am not missing something:

import sys
import random
import datetime as dt

import numpy as np
import torch

torch.manual_seed(42)
torch.cuda.manual_seed(42)
np.random.seed(42)
random.seed(42)
torch.backends.cudnn.deterministic = True

features = torch.randn(2, 5)

# Print stuff.
fnp = features.view(-1).numpy()

print("Time: {}".format(dt.datetime.now()))
for el in fnp:
    print("{:.20f}".format(el))

print("Python: {}".format(sys.version))
print("Numpy: {}".format(np.__version__))
print("Pytorch: {}".format(torch.__version__))

Run 1:

$ python tmp.py 
Time: 2019-04-13 18:23:34.530718
0.33669036626815795898
0.12880940735340118408
0.23446236550807952881
0.23033303022384643555
-1.12285637855529785156
-0.18632829189300537109
2.20820140838623046875
-0.63799703121185302734
0.46165722608566284180
0.26735088229179382324
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55) 
[GCC 7.3.0]
Numpy: 1.16.2
Pytorch: 1.0.0

Run 2:

$ python tmp.py 
Time: 2019-04-13 18:24:03.397040
0.33669036626815795898
0.12880940735340118408
0.23446236550807952881
0.23033303022384643555
-1.12285637855529785156
-0.18632829189300537109
2.20820140838623046875
-0.63799703121185302734
0.46165722608566284180
0.26735088229179382324
Python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55) 
[GCC 7.3.0]
Numpy: 1.16.2
Pytorch: 1.0.0

[CDitzel]

this is messed up. Taking all the described precautions I cannot get pytorch to exhibit deterministic behaviour or rather reproducibility. This is really frustrating

[sbelharbi]

@CDitzel did you use the script above #7068 (comment) ? or you are talking in general?