ClearAll w1 = A1*Cos[\[Beta]*x] + A2*Sin[\[Beta]*x] + A3*Cosh[\[Beta]*x] +    A4*Sinh[\[Beta]*x] w2 = B1*Cos[\[Beta]*x] + B2*Sin[\[Beta]*x] + B3*Cosh[\[Beta]*x] +    B4*Sinh[\[Beta]*x] eq1 = FullSimplify[(w1 /. x -> 0)] eq2 = FullSimplify[((D[w1, {x, 2}]) /. x -> 0)] const = Solve[{eq1 == 0, eq2 == 0}, {A1, A3}] w1 = w1 /. const eq3 = FullSimplify[(w2 /. x -> 1)] eq4 = FullSimplify[((D[w2, {x, 2}]) /. x -> 1)] (*Compalability conditions*) eq5 = (w1[[1]] /. x -> z) - (w2 /. x -> z) eq6 = (D[w1[[1]], {x}] /. x -> z) - (D[w2, {x}] /. x -> z) eq7 = (D[w1[[1]], {x, 2}] /. x -> z) - (D[w2, {x, 2}] /. x -> z) eq8 = (D[w1[[1]], {x, 3}] /. x -> z) - (D[w2, {x, 3}] /. x -> z) R = FullSimplify[   Normal@CoefficientArrays[{eq3, eq4, eq5, eq6, eq7, eq8}, {A2, A4,        B1, B2, B3, B4}][[2]]] MatrixForm[R] MatrixRank[R] b = {{0}, {0}, {0}, {0}, {0}, {0}, {0}, {1}} MatrixForm[b] (*Reduce[eq3\[Equal]0&&eq4\[Equal]0&&eq5==0&&eq6\[Equal]0&&eq7\[Equal]\ 0&&eq8\[Equal]1,{A2,A4,B1,B2,B3,B4},Reals]*) xx = FullSimplify[Inverse[R]] Det[R]; (*LinearSolve[R,b]*) Solve[{eq3 == 0, eq4 == 0, eq5 == 0, eq6 == 0, eq7 == 0,    eq8 == 1}, {A2, A4, B1, B2, B3, B4}, Integers,   MaxExtraConditions -> 2] 

Now I am able to solve the system ax=b, but I am not getting the unique solution.How to simplify to get that?. I used matrix inverse to get the x but seems like its giving all vectors which satisfy the ax=b. And also I don’t know why linear solve is not working on this problem.