Fifth Avenue Anarchy

The 1863 Orphan Asylum Burning

AI AUDIO OVERVIEW

Introduction: The Volcano Under the City

On the humid afternoon of July 13, 1863, the metropolis of New York City, the commercial heart of the Union, ceased to function as a civil society and descended into a state of primal anarchy. While the Union Army was licking its wounds and counting the dead in the immediate aftermath of the pivotal victory at Gettysburg, a different kind of battle was erupting on the streets of Manhattan. The New York City Draft Riots, which would rage for four blood-soaked days, represent the largest civil insurrection in American history outside of the Civil War itself. It was a cataclysm born of a “perfect storm” of industrialization, immigration, political corruption, and deeply entrenched racial animosity.

Within this panorama of violence, where telegraph lines were cut, mansions ransacked, and Black men lynched from lampposts, one event stands out as the singular emblem of the mob’s moral disintegration: the sacking and burning of the Colored Orphan Asylum. Located on the prestigious thoroughfare of Fifth Avenue, occupying the block between 43rd and 44th Streets, the Asylum was not merely a building; it was a sanctuary for the most vulnerable population in the city—Black children left destitute by the ravages of poverty, disease, and slavery. Its destruction was not an accidental consequence of general rioting; it was a targeted, deliberate act of racial cleansing, driven by a logic that sought to erase the presence of the African American community from the city’s landscape.

This report provides an exhaustive historical reconstruction of the attack on the Colored Orphan Asylum. It synthesizes archival records, contemporary newspaper accounts, and modern historical scholarship to offer a minute-by-minute timeline of the event. Beyond the chronology, it explores the deep structural fissures of antebellum New York that made such an atrocity possible, examining the intersection of Irish-immigrant labor anxieties, Democratic Party politics, and the volatile implementation of the federal draft. It details the heroism of the Asylum’s staff, the complex actions of the Volunteer Fire Department, and the perilous escape of 233 children through a city that had suddenly turned against them.

Part I: The Antebellum Crucible — Social and Political Context

To comprehend the ferocity of the mob that gathered on Fifth Avenue, one must first understand the social stratigraphy of New York City in the summer of 1863. The city was described by contemporaries as a “volcano under the city,” a geological metaphor for a society where immense pressure was building beneath a fragile crust of order.

1.1 The Demographic Powder Keg

By the mid-19th century, New York City was undergoing a radical demographic transformation. The Great Famine in Ireland (1845–1852) had sent waves of desperate immigrants across the Atlantic. By 1860, New York City had the largest Irish population of any city in the world outside of Dublin. These immigrants, often arriving with nothing, crowded into the squalid tenements of the Five Points and the shantytowns that dotted the upper reaches of the island.

This influx created intense friction in the labor market. The Irish immigrants found themselves on the lowest rung of the white social ladder, competing directly with the city’s free Black population for unskilled labor jobs—stevedores, porters, waiters, and domestic servants. This economic competition was exacerbated by a cultural and political environment that encouraged racial animosity. The Democratic Party machine, Tammany Hall, actively cultivated the Irish vote by framing the Republican Party and the abolitionist movement as existential threats to the white working class. The narrative was simple and incendiary: emancipation would result in a flood of freed slaves moving North, further depressing wages and displacing white workers.

1.2 The Copperhead Press and Anti-War Sentiment

The political atmosphere in New York was uniquely hostile to the Lincoln administration. The city’s commercial elite had deep financial ties to the cotton South; before the war, New York shippers controlled the export of Southern cotton, and Southern planters vacationed in New York hotels. These “Copperhead” Democrats—Northerners who sympathized with the South or opposed the war—controlled influential newspapers like the New York Daily News, the Day Book, and the Freeman’s Journal.

In the months leading up to July 1863, these papers maintained a steady drumbeat of anti-war and anti-Black rhetoric. They attacked the Emancipation Proclamation (issued January 1, 1863) as a betrayal of the war’s original purpose—preserving the Union—and framed the conflict as a “crusade for the Negro” at the expense of the white man. Governor Horatio Seymour, a Democrat elected in 1862, famously questioned the constitutionality of the draft and the war measures, lending official legitimacy to the growing dissent.

1.3 The Conscription Act of 1863

The spark that ignited this volatile mixture was the federal Enrollment Act, passed by Congress in March 1863. As voluntary enlistments plummeted and the slaughter of the war continued, the federal government instituted the first national draft. The law was invasive, requiring federal provost marshals to go door-to-door to enroll men aged 20 to 45.

However, it was the specific provisions of the law that enraged the working class. The act included a “Commutation Clause” that allowed a drafted man to avoid service by either providing a substitute or paying a fee of $300.

Table 1: The Economics of the Draft in 1863

Provision

Mechanism

Economic Reality for the Worker

Draft Liability

All male citizens and unintended citizens aged 20-45.

Universal liability in theory, but selective in practice.

Substitution

A draftee could hire another man to serve in his place.

The cost of a substitute rose with demand, accessible only to the wealthy.

Commutation Fee

A draftee could pay $300 to the government to be exempted from that specific draft round.

$300 represented nearly a year’s wages for an unskilled laborer (avg. $500/year).

Impact

Created a two-tier citizenship based on wealth.

The war was branded “A rich man’s war and a poor man’s fight”.

Source

For the Irish longshoreman or the German bricklayer, $300 was an impossible sum. The law effectively told them that their lives were expendable, while the wealthy could buy their safety. This class resentment was immediately racialized. The Democratic press argued that white men were being forced to die to free Black slaves, who would then come north to take their jobs. This twisted logic—”we are sold for $300 whilst they pay $1000 for negroes”—became the rallying cry of the mob.

Part II: The Institution of Hope — The Colored Orphan Asylum

To understand the magnitude of the tragedy on July 13, one must appreciate the institution that was destroyed. The Colored Orphan Asylum (COA) was more than a charitable organization; it was a monument to Black agency, white philanthropic abolitionism, and the progressive ideals of the era.

2.1 Founding and Mission (1836-1843)

The Association for the Benefit of Colored Orphans was founded in 1836 by three Quaker women: Anna Shotwell, infinite Shotwell, and Mary Murray. In the early 19th century, New York’s social safety net was strictly segregated. White orphanages refused to accept Black children. As a result, destitute Black children were often housed in prisons, almshouses, or left to beg on the streets.

The founders began their work by rescuing 11 children from the cellar of the almshouse. Because no coachman would transport Black children, the women were forced to walk them to their first modest home on 12th Street. The mission was two-fold: to provide immediate physical shelter and to provide moral and vocational education that would allow these children to function as free citizens—a radical proposition in a country where slavery was still the law of the land in the South and a recent memory in New York (abolished in 1827).

2.2 The Fifth Avenue Edifice

By 1843, the Association had outgrown its smaller quarters. Through tireless fundraising—often conducting “fairs” to sell handmade goods because large-scale philanthropy was rarely directed toward Black causes—the Association raised the funds to build a permanent home.

The new Asylum was a formidable structure located on Fifth Avenue, spanning the block between 43rd and 44th Streets.

Structure: It was a four-story brick building with two three-story wings, designed to be imposing and permanent.
Capacity: It could house between 600 and 800 children, though the population fluctuated. On the day of the riots, records indicate 233 children were in residence.
Value: The building was erected for $20,000 in 1843. By 1863, due to inflation and the development of the city, the real estate and the structure represented a significant accumulation of wealth, estimated at the time of destruction to be worth significantly more (approx. $650,000 in modern value).
Symbolism: In 1863, Fifth Avenue was the spine of New York’s wealth. For a “Colored Asylum” to occupy such prime real estate was a source of pride for the Black community and the abolitionists, but a source of deep resentment for the white working class, who viewed it as a “palace” for Black children while their own children lived in squalor.

2.3 Life Inside the Asylum

The Asylum was a progressive institution for its time, largely due to the influence of its medical director, Dr. James McCune Smith.

Dr. James McCune Smith: The first African American to hold a medical degree in the United States (having been forced to study in Glasgow due to racism in American medical schools), Dr. Smith was a giant of the Black intellectual community. He served as the Asylum’s physician for nearly 20 years. Under his guidance, the Asylum provided medical care that was superior to what most working-class white children received.
Education and Labor: The children received schooling in reading, writing, and geography. Upon reaching age 12, they were typically indentured—boys to farms and trades, girls to domestic service—to ensure they had a livelihood.
Demographics: While called an “orphan” asylum, the records indicate that less than a third were full orphans. Many had one parent who, due to the extreme economic marginalization of Black New Yorkers, could not afford to feed them. The Asylum functioned as a boarding school and safety net for the struggling free Black community.

By July 1863, the Asylum was a well-ordered, well-stocked institution, representing the stability and upward mobility that the rioters felt was being denied to them.

Part III: The Gathering Storm — Saturday, July 11 to Monday Morning, July 13

The destruction of the Asylum was the climax of a sequence of events that began two days earlier.

3.1 Saturday, July 11: The First Drawing

The Provost Marshal for the Ninth District, Captain Charles Jenkins, began the draft lottery on Saturday morning at his office on Third Avenue and 46th Street. The drawing proceeded peacefully, largely because the full impact had not yet sunk in. The crowd even joked as names were drawn (“Poor Jones!”), but the underlying tension was palpable. The decision to hold the draft on a Saturday was a tactical error; it gave the working men of the city all of Sunday—a day of idleness and drinking—to stew over the results and the implications of the law.

3.2 Sunday, July 12: The Fermentation

Throughout Sunday, the city’s saloons and tenements buzzed with anger. Agitators, some possibly connected to Southern sympathies or local political machines, moved through the crowds, reinforcing the narrative that the draft was unconstitutional and that the $300 clause was a deliberate oppression of the poor. By Sunday night, the police had received warnings of a plot to stop the draft, but Superintendent John Kennedy believed the city’s police force could handle any disturbance.

3.3 Monday Morning, July 13: The Explosion

6:00 AM: The uprising began. Workers at major industrial sites—shipyards, ironworks, and construction zones—failed to report for work. They formed columns and marched uptown, forcing other workers to join them, shutting down the city’s infrastructure.
10:00 AM: The draft resumed at the Ninth District office (3rd Ave & 47th St).
10:30 AM: The Black Joke Engine Company No. 33, a volunteer fire company, arrived. Their chief had been drafted, and their arrival signaled the beginning of the violence. They did not come to put out fires but to start them. A paving stone was hurled through the window of the draft office. The mob stormed the building, smashed the draft wheel, destroyed the records, and set the building on fire.
11:00 AM – 12:00 PM: The violence escalated. Superintendent Kennedy, arriving to assess the situation, was recognized and beaten nearly to death by the mob. A detachment of the Invalid Corps (wounded veterans) was surrounded, beaten, and forced to retreat, leaving their dead on the street.

As the police lost control of the Upper East Side, the mob’s focus shifted. Having destroyed the symbol of the draft (the office), they turned their attention to the “cause” of the draft: the Black population and their perceived benefactors.

Part IV: The Siege of Fifth Avenue — Detailed Timeline of the Attack

By the afternoon of July 13, the riot had metastasized. It was no longer an anti-draft protest; it was a race war. The mob, numbering in the thousands, moved west toward Fifth Avenue, drawn to the Colored Orphan Asylum as a symbol of their grievances.

4:00 PM: The Arrival of the Mob

A crowd estimated at 3,000 strong gathered around the Asylum at 43rd Street and Fifth Avenue. This mob was not a military formation but a chaotic mass of men, women, and children, armed with clubs, paving stones, axes, and torches.

The Chants: The air was filled with racial epithets. Cries of “Burn the nigger’s nest” and “Down with the Lincolnites” were reported by witnesses.
The Psychological State: The mob was driven by a belief that the children inside were living in luxury at the expense of the white worker. The well-maintained grounds and the size of the building fed into the conspiracy theory of “Black privilege” engineered by Republican abolitionists.

4:05 PM: The Initial Breach

The rioters did not hesitate. Using axes and sledgehammers, they attacked the heavy front doors of the Asylum. Windows were smashed with cobblestones torn from the street. The sound of splintering wood and shattering glass alerted the staff inside to the imminent danger.

4:10 PM: The Evacuation Strategy

Inside the building, Superintendent William E. Davis and Head Matron Jane McClellan acted with remarkable coolness under pressure. They realized that defending the building was impossible. Their only priority was the lives of the 233 children.

The Children: The children were engaged in their afternoon routine—some in the schoolrooms, some playing in the nursery, and others in the hospital wing.
The Escape Route: While the mob was battering down the front doors on Fifth Avenue, Davis and McClellan marshaled the children toward the rear exits. These exits likely opened onto the large garden grounds that extended toward Madison Avenue or the side streets (43rd/44th), which were less choked by the mob than the main avenue.
The Nature of the Escape: Some romanticized accounts suggest the mob “parted” to let the innocent children through. However, historical analysis suggests a more chaotic reality. The mob at the front was focused on breaching and looting the building. The children likely escaped because the rioters were more interested in the property than the people at that precise moment. The children were “permitted to proceed unmolested” largely because the mob was distracted by the prospect of plunder.

4:15 PM – 4:45 PM: The Defense of Chief Decker

While the children fled out the back, a dramatic confrontation occurred at the front. Chief Engineer John Decker of the NYC Fire Department arrived with Hook and Ladder Company No. 2. The role of the fire department during the riots was complex; some companies (like Black Joke No. 33) were rioters, while others (like Decker’s) attempted to do their duty.

First Fire: The rioters gained entry and set a fire on the first floor. Decker, rushing in with a handful of men, extinguished it.
Second Fire: The mob, infuriated by this interference, set fires in three different locations within the building. Decker and his men again managed to extinguish them.
The Confrontation: The mob turned on Decker. They threatened his life. In a desperate attempt to save the building, Decker stood on the front steps and addressed the crowd of 2,000 rioters. His speech is recorded in historical accounts:”He begged of them to do nothing so disgraceful to humanity and to the City of New-York to destroy a building whose only purpose was benevolence, and which had for its object nothing but good.”
The Collapse of Defense: The speech failed. The mob, now “half-drunken” and bloodthirsty, reignited the building. When they moved to physically attack Decker, his firemen surrounded him, declaring the mob would have to pass “over their dead bodies” to take him. The firemen saved their Chief, but they could not save the Asylum.

4:30 PM – 5:30 PM: Looting and Conflagration

With the defenders neutralized, the sacking of the Asylum began in earnest.

The Looting: The looting was total. The mob stripped the building of furniture, bedding, and food. Reports highlight the participation of women in the mob, who were seen carrying away the “little garments” and baby clothes that had been donated for the orphans. This detail highlights the desperate poverty of the rioters—they were stealing clothing for their own children while burning the home of others.
The Fire: By 5:00 PM, the building was a mass of flames. Within an hour and a half, the roof collapsed, and the structure was reduced to a shell. The destruction was so complete that observers noted “scarcely one brick was left upon another”. The loss of property was estimated at $35,000 (1863 currency), a massive sum for a charitable organization.

5:00 PM: The Struggle in the Streets — Paddy McCaffrey’s Intervention

While the main body of children had escaped, the chaos of the street meant that safety was not guaranteed. A group of about twenty children became separated or were surrounded by a segment of the mob that had turned away from the burning building.

The Threat: The rioters shouted murderous threats: “Wring the necks of the d—d Lincolnites”.
The Rescue: In a notable counter-narrative to the “Irish rioter” stereotype, a young Irishman named Paddy McCaffrey, along with four stagecoach drivers from the 42nd Street line and members of Engine Company No. 18, physically intervened. They fought back the mob and shielded the terrified children, escorting them to safety. This incident underscores that the riots were not a monolithic expression of Irish racism, but a complex event where individual choices defined the outcome.

Part V: The Sanctuary and the Aftermath — The 20th Precinct and Blackwell’s Island

The escape from the burning building was only the beginning of the children’s ordeal. They were now refugees in a war zone.

5.1 The 20th Precinct Station House

The staff and the rescuers led the children to the 20th Precinct Station House, located on 35th Street near 7th Avenue. (Note: Some contemporary accounts refer to this as the “35th Precinct,” likely confusing the street number with the precinct number, but official police records confirm the 20th Precinct was located on 35th Street ).

The Siege: The station house became a fortress. For three days, the children remained huddled inside, sleeping on the floor. Food was scarce. The police were forced to defend the station against repeated threats from the mob, who knew the children were inside.
The “William H. Judson” Entry: The terror of this experience is captured in the Asylum’s surviving records. An entry in the admission log for a child named William H. Judson reads: “Admitted Jul. 6, 1863. Left with us just after the riot.” This child had entered the asylum only a week before it was destroyed, only to be forced to flee for his life.

5.2 The Exodus to Blackwell’s Island

By Thursday, July 16, federal troops fresh from the battlefield of Gettysburg began to arrive in the city. The presence of the military allowed the authorities to move the most vulnerable populations to safety.

The Transport: Under the guard of Police and Zouaves (military units), the children were marched to the East River. They were ferried to Blackwell’s Island (now Roosevelt Island).
The Refugee Camp: Blackwell’s Island became a makeshift refugee camp for the city’s Black population. Thousands of Black New Yorkers, whose homes had been burned or who feared lynching, gathered there for protection under the guns of the Union navy and the shelter of the city’s almshouses and hospitals.

5.3 The Casualties

Miraculously, no children died during the attack on the Asylum itself. This is often cited as the one “bright spot” in the riots. However, the psychological trauma was immense. Furthermore, the broader riot claimed the lives of over 100 people, mostly Black men who were lynched, burned, or beaten to death in the streets.

Part VI: Analytical Insights — Why the Asylum?

The destruction of the Colored Orphan Asylum was not random. It was a highly symbolic act that reveals the core psychological and social drivers of the Draft Riots.

6.1 The Symbolism of “Black Privilege”

To the starving Irish laborer, the Colored Orphan Asylum represented a perverse inversion of the natural order. They saw a grand, brick “palace” on Fifth Avenue, funded by wealthy white abolitionists, where Black children were fed, clothed, and educated. Meanwhile, the Irish drafted soldier left behind a family with no safety net, facing starvation.

Insight: The attack was an act of “leveling.” By destroying the Asylum, the mob was not just attacking Black children; they were attacking the white philanthropy that they believed elevated Black interests above those of the white working class. The looting of food and clothing confirms that economic envy was as potent a motivator as racial hatred.

6.2 The Failure of the City

The event highlighted the catastrophic failure of New York City’s civic infrastructure. The police were understaffed and overwhelmed. The militia was absent. The fire department was divided against itself. The fact that the safety of 233 children relied on the quick thinking of a Matron and the courage of a single Fire Chief (Decker) rather than a coordinated state response indicts the city’s preparedness and its political leadership, which had flirted with insurrectionist rhetoric for months.

6.3 The End of an Era

The burning of the Asylum marked the end of a specific chapter of Black life in Manhattan. The Asylum never rebuilt on Fifth Avenue. The land was too valuable, and the racial hostility too intense.

Relocation: The institution moved first to a temporary home in Carmansville (West 150th St), then to 143rd Street in Harlem, and finally to Riverdale in the Bronx.
Migration: This mirrored the broader migration of the Black community. The violence of 1863 spurred a mass exodus of African Americans from Manhattan, many moving to Brooklyn (Weeksville) or New Jersey to escape the violence. The riots effectively cleansed the mid-town area of its Black population.

Conclusion

The attack on the Colored Orphan Asylum remains one of the most harrowing episodes in American urban history. It serves as a stark reminder of how quickly the veneer of civilization can crack under the pressures of war, poverty, and racism. The image of Chief Decker standing alone on the steps, pleading with the mob in the name of “humanity,” stands in contrast to the image of the mob looting baby clothes from a burning orphanage.

The survival of the 233 children is a testament to the courage of a few individuals—Davis, McClellan, Decker, and McCaffrey—who held onto their humanity when the city around them had lost theirs. However, the ashes of the Asylum on Fifth Avenue signaled a permanent scar on the city’s history, changing the geography of Black New York forever and leaving a legacy of racial tension that would persist for generations.

Table 1: The Economics of the Draft in 1863

Metric

Data

Date

Approx. 4:00 PM

Time of Attack

Approx. 4:00 PM

Location

Fifth Avenue, between 43rd and 44th Streets

Building Value

$35,000 (1863 currency)

Population

233 Children

Casualties (Asylum)

0 Deaths (All evacuated)

Mob Size

3,000+ (men, women, children)

Rescuers

Supt. Davis, Matron McClellan, Chief Decker, Paddy McCaffrey

Source | Source | Source | Source | Source

The Colored Orphan Asylum was destroyed, but the institution survived, evolving into the Riverdale Children’s Association. Its survival is a testament to the resilience of the community it served, even in the face of the “volcano” that erupted in July 1863.