What happens when machines surpass the most brilliant human minds? Mathematicians are finding out