Muchan, the given problem is wrong. (first part, i didnt try the second part)
Lets take teta = 0, then sin() = 0 cos() = 1
=> a = 1, b = 2
LHS: (a-1)^2 = 0
RHS: 5 - 2b = 1
LHS not equal RHS.
Please check the problem again.
BTW, sinhalen liyala tiyenne monawada, i dont have sinhala fonts mucho.