Following passage of the legislation establishing the Free Academy, the selection of a site on which to construct a school building ensued. A proposed site on Madison Square was met with opposition from those who did not want the public school intruding on the upper-class neighborhood developing there. A nearby site at the southeast corner of Lexington Avenue and 23rd Street was selected instead. The Free Academy was designed in the Gothic style favored for academic buildings by James Renwick, who would go on to design St. Patrick’s Cathedral. The four-story building was constructed primarily of brick covered with brown stucco to resemble stone. It included eighteen classrooms and a main assembly hall referred to as the “chapel.” The campus consisted of a small yard, separated from 23rd Street and Lexington Avenue by an iron fence, and an alley between the school and adjacent buildings on 22nd Street.
When it opened in 1849, the Free Academy, located at the upper reaches of the developed part of the city, was surrounded by much vacant land. Richard Bowker, Class of 1868, reminisced about being able to "look across green fields southward, over Gramercy Park; westward, beyond Madison Square; northward, to Rose Hill…with its few houses; and eastward clear to the East River." Students used to walk home to the Lower East Side across vacant lots and "steal away for half hour for a swim in the unfrequented river." Another alumnus, James Steer, Class of 1853, remarked upon the "primitive state of things" that allowed groups of students to play games in the middle of 23rd Street "with no one to forbid, no policeman in sight, few, if any houses, in the neighborhood."
