Fort Harrison was an important component of the Confederate defenses of Richmond during the American Civil War. Named after Lieutenant William Harrison, a Confederate engineer, it was the largest in the series of fortifications that extended from New Market Road to the James River that also included Forts Hoke, Johnson, Gregg, and Gilmer. These earthworks were designed to protect the strategically important Chaffin's Bluff on the James.

On September 29, 1864, 2,500 Union soldiers from Maj. Gen. Benjamin Butler's Army of the James overran Major Richard Cornelius Taylor's 200-man Confederate garrison and captured the fort in the Battle of Chaffin's Farm. Brig. Gen. Hiram Burnham, a native of Maine and a brigade commander in XVIII Corps, was killed in the assault, and the Union-held fort was renamed Fort Burnham in his honor.

Although the attacks of September 29 had succeeded in capturing only Fort Harrison, General Robert E. Lee saw the potential threat to Richmond and ordered a counterattack on September 30. The attack failed, but Brig. Gen. George J. Stannard lost an arm while resisting Lee's assault. This failure forced the Confederates to realign their defenses farther west. Fort Burnham remained in Union hands until the end of the war.

In 1930, members of the Richmond Parks Corporation, a local preservation society, constructed a log cabin on the site to serve as their headquarters. Today, this building serves as the Fort Harrison visitor center, part of Richmond National Battlefield Park.