Privacy-Preserving Restricted Boltzmann Machine

Algorithm 3

Privacy-Preserving Distributed Algorithm for RBM.

Initialize

all weights (𝑊,𝑏,𝑐) to small random numbers and make them known to both parties.

Repeat

for all training sample {𝑣⁰_𝐴,𝑣⁰_𝐵} do

⧵⧵ This part mainly uses (5) and (9). Samples are obtained after only one step of Gibbs sampling

because one-step Gibbs has been shown to work surprisingly well [22].

Step 1. Gibbs Sampling

{

 For each hidden layer node ℎ⁰_𝑗, Party 𝐴 computes ∑_{𝑘≤𝑚𝐴}𝑤_𝑗𝑘𝑣⁰_𝑘+𝑐_𝑘, and Party 𝐵 computes ∑_{𝑚𝐴≤𝑘≤𝑚𝐴+𝑚𝐵}𝑤_𝑗𝑘𝑣⁰_𝑘+𝑐_𝑘

 Using Algorithm 1, Parties 𝐴 and 𝐵 jointly compute the sigmoid function for ℎ⁰_𝑗, obtaining the random shares ℎ⁰_𝑗1 and ℎ⁰_𝑗2,

 respectively, s.t. ℎ⁰_𝑗1+ℎ⁰_𝑗2=𝑓∑_𝑘𝑤_𝑗𝑣⁰_𝑘+𝑐_𝑘

 For each visible layer node 𝑣¹_𝑖, after that Party 𝐴 computes ∑_{𝑘≤𝑚𝑛}𝑤_𝑖𝑘ℎ⁰_1𝑘+𝑏_𝑘, and Party 𝐵 computes ∑_{𝑘≤𝑚𝑛}𝑤_𝑖𝑘ℎ⁰_1𝑘+𝑏_𝑘

 then again using Algorithm 1, Parties 𝐴 and 𝐵 jointly compute the sigmoid function for 𝑣¹_𝑖,

 obtaining the random shares 𝑣¹_𝑖1 and 𝑣¹_𝑖2, respectively, s.t. 𝑣¹_𝑖1+𝑣¹_𝑖2=𝑓∑_𝑘𝑤_𝑖ℎ⁰_𝑘+𝑏_𝑘

 For each hidden layer node ℎ¹_𝑗, Party 𝐴 computes ∑_{𝑘≤𝑚𝐴}𝑤𝑣¹_𝑘+𝑐, and Party 𝐵 computes ∑_{𝑚𝐴≤𝑘≤𝑚𝐴+𝑚𝐵}𝑤𝑣¹_𝑘+𝑐

 Using Algorithm 1, Parties 𝐴 and 𝐵 jointly compute the sigmoid function for ℎ¹,

 obtaining the random shares ℎ¹₁ and ℎ¹₂, respectively, s.t. ℎ¹₁+ℎ¹₂=𝑓∑_𝑘𝑤𝑣¹_𝑘+𝑐

}

return   𝑣⁰₁,𝑣⁰₂ and ℎ¹₁,ℎ¹₂ to two parties.

⧵⧵ This part mainly uses (4), (6) and (7).

Step 2. Update Weights

{

 Parties 𝐴 and 𝐵 compute 𝑣⁰₁ℎ⁰₁, 𝑣¹₁ℎ¹₁, 𝑣⁰₂ℎ⁰₂ and 𝑣¹₂ℎ¹₂ respectively.

 Parties 𝐴 and 𝐵 apply Algorithm 2 to securely compute the product 𝑣⁰₁ℎ⁰₂,

 obtaining random shares 𝑟⁰₁₁ and 𝑟⁰₁₂, respectively,

s.t. 𝑟⁰₁₁+𝑟⁰₁₂=𝑣⁰₁ℎ⁰₂. Similarly, they compute the random partitions of 𝑣⁰₂ℎ⁰₁, 𝑟⁰₂₁, and 𝑟⁰₂₂, s.t. 𝑟⁰₂₁+𝑟⁰₂₂=𝑣⁰₂ℎ⁰₁.

 Parties 𝐴 and 𝐵 apply Algorithm 2 to securely compute the product 𝑣¹₁ℎ¹₂,

 obtaining random shares 𝑟¹₁₁ and 𝑟¹₁₂, respectively,

s.t. 𝑟¹₁₁+𝑟¹₁₂=𝑣¹₁ℎ¹₂. Similarly, they compute the random partitions of 𝑣¹₂ℎ¹₁, 𝑟¹₂₁, and 𝑟¹₂₂, s.t. 𝑟¹₂₁+𝑟¹₂₂=𝑣¹₂ℎ¹₁.

 If Party 𝐴 owns the visible unit, which is 𝑣⁰₁, then Party 𝐴 computes 𝑣⁰₁ℎ⁰₁+𝑟⁰₁₁−𝑣¹₁ℎ¹₁−𝑟¹₁₁−𝑟¹₂₁,

 and Party 𝐵 computes 𝑟⁰₁₂−𝑣¹₂ℎ¹₂−𝑟¹₁₂−𝑟¹₂₂. Then Party 𝐵 sends this value to Party 𝐴,

 who can add these two numbers to get the log-likelihood gradients

−(𝜕log𝑝(𝑣)/𝜕𝑊_𝑖𝑗) for an RBM.

If Party 𝐵 owns the visible unit, then the same method can be used to calculate the value.

 Then using these log-likelihood gradients Party 𝐴 can update the parameter 𝑊 of this RBM.

𝑤_new=𝑤−𝜂(𝜕log𝑝(𝑣)/𝜕𝑊_𝑖𝑗)

}

return  new 𝑊 to two parties

Using the same method, we can update parameters 𝑏 and 𝑐

end for

Until (termination condition)