THE SOUTHERN LOYALISTS

By: Robert E. Hurst

So you think that just because you come from the south, your ancestors must have been Confederates? Richard Nelson Current, author of "Lincoln's Loyalists," estimates that as many as 100,000 white, Southern males wore the Blue instead of the Gray as regular soldiers or local militia. Did you know every state in the Confederacy except for South Carolina raised at least one unit for the Federal Army, for a total of 55 regiments?

Were these men patriots or were they, as some Southern historians such as William Stanley Hoole maintained, traitors to the Cause? Who were these men who risked so much to remain loyal to "the Old Flag, sealed with the blood of our forefathers?" Were they uneducated hill folk, or were they sophisticated anti-slavery and anti-plantation visionaries? I want to address the Southern Unionist, specifically,

1. The facts of their numbers and demography.

2. An assessment of their impact on the conduct of the war.

3. Some history of one of the most valiant of these units, the First Alabama Cavalry, US Volunteers and one soldier in it, the 50-year old Pvt. Billington Sanders Hurst.

4. The fate of these men after the war.

From the perspective of the present, we must deal with the Myth of the Lost Cause rather than the reality. With time, the myth has grown to epic proportions; a solid Confederacy of people standing firm against the tyranny of the strong central government, of agriculturalism and Jeffersonian Democracy against the factory and the political machine. From the very beginning, division haunted the Confederacy. In Alabama, for example, of 52 counties, 23 voted to remain within the Union. These counties lay mostly within the Appalachian Highlands of northern Alabama, and this division was one that was repeated everywhere within the South. The people of the uplands were pro-Union, and the people of the plantation areas represented the fire-eaters of the Confederacy. Where plantations ruled, slavery held sway, and the Confederacy was strong. Where free men tilled the soil, slavery, slaves, the Confederacy, and most particularly the Planters, were not popular. On the coastal lowlands of North Carolina, where the people earned their living by fishing and commerce, lumbering and other enterprises that were inimical to the slave labor economy, the people tended to remain loyal to the nation their forefathers had fought to build.

Sadly, the war that was fought among Southerners was never civil, and in a war characterized by grand gestures among the principal armies, viciousness seemed to prevail. We're all familiar with Grant stopping the war to present his friend Pickett with a silver tea set on the occasion of the birth of his child. But how many know that same George Pickett faced prosecution as a war criminal for hanging 22 men of the First NC. Hostage taking and murder were common, and the Confederates so feared the Unionists within their midst that they used brutal tactics to suppress them.

Consider Alabama, the only state for which I have the figures handy. A Deep South state, the location of the first capitol of the Confederacy, yet 23 counties voted the "Cooperationist" ticket. In Winston County, not a single vote was cast for secession. In the 23 loyalist counties, the vote was 21,665 to 12,042. However, the plantation states wielded the power, and in the slave-holding plantation counties, 24,865 voted to secede and 6,965 voted to remain. The totals for the state were 28,630 to remain in the Union and 36,907 for secession. Barbour, Bibb, Butler, Henry, Lowndes, Marengo, Pike and Russell Counties tallied no Cooperationist votes. Thus, 43.7% of the voters of the ostensibly solid Confederate state voted to remain within the Union.

The vote for secession followed closely the distribution of slaves or the number of bales of cotton produced in Alabama. Winston County held only 122 slaves, or 3.41% of the population, and in most of the loyal counties, the proportion of slaves was less than 20%. However, substantial Unionist sentiment was found even in the plantation counties. In Green County, with 76.5% of the population being African slaves, nearly 40% of voters wished to remain within the Union. It must be pointed out that these figures overestimate Unionist support because many Southerners voted Cooperationist and then enlisted in Confederate forces. Nonetheless, I think we can conclude that at the beginning, the South was far from united in its rebellion, and substantial minorities still held the Old Flag in high regard, while in some areas the loyalists comprised a majority. This represented a worrisome threat that drained substantial Confederate resources throughout the entire war.

The mountain area in western Virginia and eastern Tennessee was one of the major hotbeds of Unionism. Virginia, arguably the very heart of the Confederacy, was split along lines similar to those seen in Alabama. The people of the mountain counties of Virginia remained so solidly Union that they petitioned to be admitted as a separate state in 1863. Tennessee was perhaps the most split of the states voting to secede and sent approximately equal numbers to each Army. As mentioned by Sam Watkins, the First Tennessee, the regiment that stood Sherman's assaults on the Dead Line at Kennesaw Mountain all alone and saying they needed no help, had Union sympathizers in its ranks.

The Unionists were a thorn in the side of the Confederacy from the beginning, and Abe Lincoln, that wily old politician, sought to exploit this advantage. East Tennessee and West Virginia were hotbeds of outrage against the rebellion. The only problem for Lincoln was to get access for recruiters and arms.

Lincoln immediately authorized a number of individuals to raise regiments of loyal Virginians. Normally the governor of the state was responsible for enrolling the state militia into the Federal Army, and clearly none of the Southern governors would be any help. William Burton, the governor of Delaware refused to cooperate, but Lincoln found a way to enroll the Delawareans through Robert Patterson, then commanding the Pennsylvania troops. The First Delaware enrolled through Pennsylvania, and at first, Lincoln used this mechanism to enroll Virginians. Ohio served as a major recruiting ground for Virginians.

Early Federal military movements revolved around protecting the Virginia Unionists and maintaining access to these military resources. McClellan moved on Grafton in May of 1961, partly to protect the B&O Highway and partly to protect the Unionists. Cox moved up the Kanawha toward Charleston, then held by Gen. Henry Wise, a rabid secessionist and ex-governor of Virginia. Wise reported to Lee he was surrounded by hostile people. "They invite the enemy, feed him, and he arms and drills them. A spy is on every hill top, at every cabin, and from Charleston to Point Pleasant they swarm." On the retreat "the State volunteers under my command lost from three to five hundred men by desertion. The Kanawha Valley is wholly disaffected and traitorous." Well, not entirely. Thomas Jackson called the Kanawha home.

The government of the State of Virginia moved its military forces to attempt to stem the hemorrhage of potential manpower and moved some 5000 men to Mill Creek, near Martinsburg, which slowed recruiting across the Ohio considerably.

A few men even were raised in the heart of the Tidewater. In spite of hopes to raise an entire regiment of Loyal East Virginians, only a single company, The First Loyal East Virginia Infantry rallied to the Old Flag. Other men joined the Accotink Home Guard. Both units spent most of their time patrolling telegraph lines. But when the total was added up, 31,872 white Virginia men, including those who joined before West Virginia was admitted as a separate state, wore the Blue. This figure did not count militia who did not enter Federal service. When it was said that on July 3, 1863, as the troops lined up for their ill-fated assault, that "All Virginia was there," it just wasn't true.

Tennessee was a bitterly divided state, and considerable efforts were expended by both sides either to deny the military resources to the enemy or to tap them. East Tennessee, particularly, was a hotbed of Union sentiment, and no one was more vigorous in the cause than Andrew Johnson. Johnson had campaigned vigorously in East Tennessee prior to the secession vote. A friend of Johnson's, James T.T. Carter, an Annapolis graduate and lieutenant in the US Navy, was detailed from the Navy to drill troops in East Tennessee. Carter, incidentally, was the only American to hold the ranks of Major General and Rear Admiral.

The problem with East Tennessee was its inaccessibility to the Federals. Guarded by the Cumberland Mountains, and cut only by three passes, the region was denied to the Union. The trip to Union territory was arduous, and the Rebels managed to interdict the passes. A New Market physician and Mexican War veteran, John W. Thornburgh, organized a cavalry company and was ambushed in Baptist Gap. Only about a third managed to get through to Barbourville, Kentucky. Thornburgh, himself, and 8 men were captured. The remainder retreated to their homes. Nonetheless, some two thousand men managed to escape and became the First and Second East Tennessee.

Reaching the loyalists was a source of contention between Lincoln and his generals. Buell saw only the military difficulties and was less than enthusiastic. George Thomas, one of his division commanders, was even less enthusiastic. Buell's refusal to move on East Tennessee was one of the main reasons Lincoln replaced him.

Another Carter brother, William B., looked to organize a bolder stroke. On the night of November 8, loyal men burned a number of bridges on the east Tennessee and Virginia Railroad, one of the few steel arteries of the South. The Confederates reacted with a savagery that belied their talk about "rights". Isham G. Harris, the fire-eating governor of Tennessee, wrote to Jefferson Davis, "The burning of railroad bridges in East Tennessee shows a deep-seated spirit of rebellion in that section. Union men are organizing. This rebellion must be crushed out instantly, the leaders arrested, and summarily punished."

Confederate troops began scouring the hills, and slowly the numbers of captives increased, reaching the hundreds. Secretary of War Judah Benjamin sent the following instructions for dealing with the "traitors".

1st. All such as can be identified as having been engaged in bridge-burning are to be tried summarily by drum-head court-martial, and, if found guilty, executed on the spot by hanging. It would be well to leave their bodies hanging in the vicinity of the burned bridges.

2nd. All such as have not been so engaged are to be treated as prisoners of war, and sent with an armed guard to Tuscaloosa, Ala., there to be kept imprisoned . . . till the end of the war.

P.S. Judge Patterson, Colonel Pickens, and other ringleaders of the same class must be sent at once to Tuscaloosa to jail as prisoners of war.

Several prisoners were hanged and left near the railroad, where passengers were encouraged to flog their dead bodies with canes as the train passed. The weather being somewhat warm, the corpses were cut down after only 36 hours. This barbarous treatment was justified by Jefferson Davis' proclamation, "stating that all those who did not fully recognize their allegiance to the Government should remove from its limits, with their effects before October, 1861. Those persons who remained tacitly recognized the Government and are amenable to the laws."

Not everyone supported such high-handed practices and recognized the resulting calm was more apparent than real. Meanwhile, the disaffected continued to trickle through the passes. Finally, after Grant opened the way by taking Forts Donelson and Henry, Nashville fell in February, 1862 and East Tennessee was available for recruiting. Andrew Johnson was appointed military governor, but he and his subordinates botched the recruiting job by bickering among themselves. Besides, some of the ardor had been cooled by the Emancipation Proclamation.

The Confederates tried to recruit in East Tennessee, finally resorting to the draft. Many wondered whether the results were worth the effort. East Tennesseans were notoriously unreliable, often surrendering at the first opportunity, or simply deserting. Worse, the more conscription was enforced, the more men went North.

The deserters presented problems to both sides. The Confederates felt justified in executing men who had left the Confederate Cause to join the Union. Yet, these men claimed the only reason they wore the Gray was because of the draft, and had they been given a choice, they would have worn the Blue in the first place. The Federal authorities recognized that they were asking a lot of men to doubly risk their lives by serving against the Confederacy. Many of the "galvanized Yankees" were placed in units that guarded the frontiers and fought Indians.

All over the South, the pattern of Tennessee was repeated. In North Carolina, a number of regiments were raised on the coast and in the highlands. Even Georgia sent a regiment near the end of the war. In Alabama, my great great grandfather, Billington Sanders Hurst managed to elude Confederate patrols and, at age 50, rode 210 miles from his home in St. Clair County, Alabama, to enlist as a private in the First Alabama Cavalry, US Volunteers. More about the proud Fighting First later.

In Arkansas, the main problem was with equipping volunteers. Surprisingly, Louisiana was also a hotbed of loyalist sentiment. The Cajun population, particularly, held no love for the planters and enlisted in Union units in considerable numbers. The Irish, German and Yankees of New Orleans saw the Confederate cause as treason, and when Butler and Farragut steamed up the Mississippi in April, 1862, the dragooned men holding Ft. Jackson were such unwilling conscripts that they spiked their guns and shot the officers who wouldn't agree to surrender. The fort fell without a Union shot being fired.

If a brigade of Federals could have worked their way through Indian Territory to West Texas and the Hill Country, Texas would undoubtedly have returned to the Union. Sam Houston, the governor at the time of the secession vote, had done everything legal and illegal he could manage to keep Texas in the Union. The German immigrants saw no advantages to the Confederacy. Even today, few of the courthouses in areas settled by Germans display the ubiquitous Confederate infantryman on the square. The Hispanic population in Texas was solidly Unionist, and a number of irregular units were formed. These units were most unreliable, however, because Mexico started its own civil war about that time, and these men had the bad habit of taking their equipment further South. The situation in Texas was particularly bitter. More than a hundred Unionists were hanged for their loyalty. Every state of the Confederacy except for South Carolina sent at least one regiment of white men to the Union Army. Mississippi contributed the First Mississippi Mounted Rifles, though the unit never filled completely. Florida contributed the First and Second Florida Cavalry. Georgia contributed the First Georgia Battalion. Many other Georgia men enlisted in Tennessee units or the First Alabama, as did nearly 100 South Carolinians and 300 Mississippi men.

PART II. Who Were These Men

Why were some men of the South vehement supporters of the Union, while other Southerners rallied to the Stars and Bars? William Stanley Hoole, author of a monograph on the First Alabama, characterized them as a "poor, often underprivileged people who had long been isolated on their rocky highlands, suspicious of intruders and generally antisocial. Blindly hating the affluent slave-holder and his slave alike, they had first refused to support the cause of secession and, afterwards, ignored all Confederate civilians and military conscription laws." The descendants of the men of the First have charged Hoole, better known for his Confederate histories, with slander. In fact, Hoole was only reporting what the Myth of the Lost Cause needed to claim. To validate the Great Rebellion, the loyalists needed to be discounted.

Very few Unionists owned slaves, but, then, very few ordinary Confederate soldiers were slave-holders, either. Throughout the South, only one family in three owned as much as a single slave. Examination of the 1860 census for Washington County, NC, which furnished roughly equal numbers of men to each army, showed the average Union soldier owned only $269 in personal property. The average Confederate soldier owned $3,759, but is this an example of the fallacy of the mean? None of the Unionist heads of household reported more than $1000 in family income, while a couple of dozen Confederates did. All but one of the Unionists was a landowner, while 19 of the Confederates were landless. After all, the mean of one million dollars and one hundred dollars is $500,050.

Close examination showed the Confederates of Washington County to be large planters and their dependents, their sons, the merchants with whom they dealt, the lawyers and clergymen they patronized, and the poor white men who worked as day laborers, an alliance of the very rich and the very poor. The division between Unionist and Secessionist was not simply between rich and poor. The middle class that had no economic interest in the slave economy tended to be solidly Unionist, and why not? These Middle-class folk themselves were oppressed economically by the system. Moreover, they tended to be mightily offended by the airs put on by the planters, who tended to see themselves as a privileged aristocracy. In the case of the Hurst family, Billington owned 160 acres of land in St. Clair County, Alabama. I've seen his property, and it is good bottom land, rich and productive, and I doubt if he were a hillbilly. Interestingly, the geographic division of politics remains still in Alabama, where politics in the northern part of the state still has a much stronger populist flavor than in the old plantation counties.

PART III. The Fighting Southern Federals

What was the impact of the Southern Loyalists? Three factors need to be considered; the direct contribution of the men as soldiers to the Union cause, the resources expended by the Confederacy to counter the threat, and finally, the loss of manpower to the Southern cause. Taking these in reverse order, the loss of manpower to the South was probably fatal to its cause. While estimates of the numbers differ. Current estimates that as many as 100,000 white men of the South served the Union cause as Federal forces and local defense forces. This was more men than Lee or any other Southern commander ever had under arms at any time. In addition, thousands of other troops were diverted from the main armies to control the loyalists. Cavalry patrols. How much difference would the cavalry patrols that tried to interdict the flow of manpower have made to the cavalry-poor army of Johnston?

Consider the impact of the 30,000 East Tennesseans who joined the Union. Had they joined the Confederate forces, this would have amounted to a swing of 60,000 men, and when the 10,000 Confederates who were required to keep East Tennessee in subjugation are added in, a difference of 70,000 men results.

Finally, there is the direct contribution. There is no question that some of the southern units were hard-fighting, crack units, while others were of questionable value. The Tennessee Unionists units were of solid quality, as were most of the Virginia units, who saw fighting almost from the beginning at Philippi and Romney under McClellan. The First Mississippi Mounted Infantry rode with Grierson in his famous raid through the heart of Mississippi. In the movie "The Horse Soldiers" with John Wayne and William Holden, the Southern-speaking men (Ken Curtis) were authentic and represented the First Mississippi. It is true that when Pickett executed 22 men of the First North Carolina USV, he did, in fact, cut the heart out of some units, particularly those containing "galvanized Yankees." Still, these men could, and did, perform valuable duty in less exposed positions.

Other Unionists such as the First Alabama were dependable units, just as hard-fighting as any Ohio, Maine, or Pennsylvania troops. The Myth of the Lost Cause demands the loyalists be branded as poor soldiers. Interestingly, many of the Unionists served in cavalry units, and early in the war, the quality of the Union cavalry in general was very poor. But by 1864, the Federal cavalry were, in general, at least equal to the Confederates. The hard-riding Blue troopers of Phil Sheridan scattered Jeb Stuart's plumed cavaliers and killed the famed cavalryman. While they never tamed "that devil Forrest", the Union cavalry in general, and the First Alabama did humble Joe Wheeler and Wade Hampton.

The First Alabama began humbly. Poorly trained and equipped, and sent against superior numbers of Confederate cavalrymen in 1862, they fared poorly early. But by the time Sherman began his Red Clay Minuet with Johnston, the First was a solid, invaluable unit. The regiment was formed in 1862 in Huntsville and Memphis and mustered into Federal service that December in Corinth, Mississippi. Company officers were chosen from among the men, and Captain George E. Spencer was later named Colonel and given overall command.

During most of its operational life, the First Alabama was part of the 16th Corps, Army of the Tennessee. As a cavalry unit, its missions were scouting, raiding, reconnaissance, flank guard and screening the army on the march The names on its battle flag, like most cavalry actions, are mostly forgotten; Nickajack Creek, Vincent's Crossroads and Cherokee Station among others. Better known names are there, too; Streight's Raid through north Alabama; and battles at Dalton, Resaca and Kenesaw Mountain in the Atlanta Campaign.

One general characterized the Alabama troops "invaluable...equal in zeal to anything we discovered in Tennessee." And Major General John Logan, commanding the 15th Army Corps in Sherman's forces, praised the troopers as "the best scouts I ever saw, and (they) know the country well from here to Montgomery." General Sherman, knowing the value of his Alabama troops as soldiers and symbols of the loyal South, honored the First by his selecting it to be his escort on the march from Atlanta to the sea.

The First, part of Kilpatrick's Third Cavalry Brigade (with the Fifth Kentucky and Fifth Ohio) rode over 700 miles in 55 days during the winter of 1864. In February, they routed a brigade of Wheeler's cavalry at Williston, SC, taking 5 battle flags and scattering the Confederates over miles of countryside. On March 10, surprised in camp by 5,000 of Wade Hampton's and Joe Wheeler's cavalry, the 800 men of the Third Brigade killed 103 of their attackers with the loss of 18 men and officers at Monroe's Crossroad (also known as the Battle of Kilpatrick's Pants). The official report said that "a bloody hand-to-hand conflict" followed, lasting more than three hours. Brave Lieutenant Stetson managed to turn the tide when he crept to one of his guns and delivered a barrage of cannister into the ranks of the attackers.

The First displayed a darker side, also and was "zealous in its chastisement of Rebels". They took seriously their role as foragers, and it was charged the First, knowing where a Southerner was likely to stash his food, never went hungry. This is not entirely true because they often went hungry and their horses starved for lack of forage. They relished their job as incendiaries, too and "laid Barnwell in ashes" despite Kilpatrick's efforts to stop them. These actions are more understandable when one learns the fates of their families. Many had their homes burned or their families abused, and some saw their kin lynched by vindicative Confederates. So badly were many of their innocent families treated that a "Refugee House" was established in Nashville for those who were able to escape Rebel persecution. Our family fared well, probably because Billington's son, my great grandfather, served in the 19th Louisiana Infantry, CSA.

PART IV. The Fate of the Southern Unionists During Reconstruction

The fate of the Southern Unionists began to be clear with the massacre at Ft. Pillow, where Forrest's men massacred a number of white soldiers of the 13th Tennessee Cavalry. The commanding officer, Maj. Bradford, was shot after he had surrendered and was being taken to Forrest's headquarters at Jackson, TN. Yet, Ft. Pillow is remembered mainly for the massacre of black soldiers. Both Congress and Lincoln were forgetting the Tennessee loyalists, and forgotten they are today. Yet, these men risked more for their nation than did the men of the North, for they risked execution on capture and consigned their families to the often not-so-tender mercies of their often unforgiving neighbors.

In part, they fell into obscurity because Lincoln saw the African-American population as representing a larger manpower resource, and after the war, the Radical Republicans sought to consolidate their power through the freed slaves rather than the Southern Unionists. There were some exceptions. Col. Spencer, commander of the First Alabama, was elected governor and then Senator, and was the only Republican re-elected to the Senate. Nontheless, he ended up his years in Nevada, leaving Alabama for a variety of reasons.

The end of the war did not end the private grudges that the division of the Southern whites had produced. Victorious Union veterans sought retribution for the depravations their families had suffered during the war. Yet, the government failed to reimburse them for their losses, or even to provide effective protection in many areas. Andrew Johnson was extremely generous in pardoning ex-Confederates, and in many areas these pardoned men established governments that were inimical to the Union veterans. Some were murdered, many left and went West, while others, such as Billington Sanders Hurst moved away from their homes. On his return, he gathered his goods and left. Our family never spoke of him again and I was the one who discovered his Union service. He moved to Jefferson County, Alabama, where he married again. In 1881 he was apparently destitute and applied for a disability pension.

Reconstruction under the Act of 1867 brought temporary relief, but the loyalists, like all Southerners, had been impoverished by the War. The Southern economy was in shambles, and the industrial powers of the North quickly established their economic hegemony. Until the late 1940's Southern goods moving north paid a higher price on the railroads than Northern goods moving South or Southern raw materials moving north. Moreover, the white loyalists felt alienated in the Republican Party, which tended to give more emphasis to the needs of the freed slaves than to the loyal whites. One of the tenets of Northern industry was to divide and conquer, and by setting black against white, a reservoir of cheap labor could be guaranteed. Additionally, the often corrupt and inefficient "Carpetbagger-Scalawag) governments did little to help Southerners of any color or loyalty, preferring to line their own pockets. Finally, as racial divisions emerged in the South, the loyalists saw how they finally would have to decide their political loyalties, and so they submerged into the white culture virtually without a trace. Only in the last decade have most descendants of men who served with the First Alabama learned the truth about their ancestors.