Pipelined Training with Stale Weights in Deep Convolutional Neural Networks

Algorithm 1

Pipelined stochastic gradient descent (PPL-SGD).

	Initialize weights
	Given learning rate sequence
	fortodo
	fortoin parallel do
	Compute gradient using stale weights:
	Update weights:
	end for
	end for