Madison, the county seat of Morgan County, was incorporated March 12, 1866. The city was named in honor of former U.S. President James Madison.

Madison has the largest designated historic district in Georgia that encompasses most of the town. During the Civil War, Sherman’s Army spared Madison destruction because it was the home of pro-Union Senator Joshua Hill.