Tauba Auerbach’s brilliant, mathematical paintings and sculptures exist in what they call a “scary place of unknowns.”