James Madison agreed that election of the people at large was the best way to go about electing the president, but he knew that the less populous slave states would not be influential under such a system, and he backed the Electoral College. Another factor here was the so-called Three-Fifths Compromise, which gave added power to the slave-holding states under the Electoral College which they would not have had under any likely form of popular vote.
https://en.wikipedia.org/wiki/Federalist_No._68