Downstairs/ Upstairs at Sylvester Manor

Face to face with slavery in the North

The manor house as it appears today: The core structure dates from ca.1740, when Brinley Sylvester replaced the original house, which had stood on the property since the 1650s. Brinley was a grandson of Nathaniel Sylvester and his wife, Grizzell, the estate’s first owners.

Sue Daley and Steve Gross

Excerpted from The Manor: Three Centuries at a Slave Plantation on Long Island, by Mac Griswold; Farrar, Straus and Giroux.Copyright © 2013 by Mac Griswold.

"Dear Mr. and Mrs. Fiske,”  I write  in July of 1984, “I rowed into the creek below your house and could not resist walking around your  lawns  and  garden,”  and I finish  by saying I am a garden historian and would like to meet them and learn about the history  of their house. Several months  and three  letters  later, I receive a reply, and an invitation to visit.

A short car-ferry ride takes me from my home in Sag Harbor, on the East End of Long Island, to  Shelter Island, and I drive to the house that so captured my imagination. I pull through the white-painted cement gates and rattle down the long wooded driveway, two worn sandy tracks separated by a grass ridge. Two small figures stand under the front portico, Andrew Fiske, in a wrinkled linen jacket and open-necked checked shirt, and Alice, his wife,  in  skirt  and  blouse and a chintz mobcap.

Portraits of Ezra L’Hommedieu (left) and his wife, Mary, hang in what had been the “hall,” the heart of the house, now dubbed the “paneled parlor.” Ezra was the great-grandson of Nathaniel Sylvester.

Sue Daley and Steve Gross
As they invite me into the darkened front hall, light hits the polished brass doorknob. A large key gleams in the heavy, square lock. I step across the threshold and into a history project spanning eleven generations, three and a half centuries, and four continents. Here, where the Hamptons, jittery playground of the rich and famous, are only eight miles away, I am astonished to learn I’m meeting a member of a family that has lived on this tranquil-seeming property in an unbroken line since 1652.

Before his death in 1992, Andrew Fiske (right) was the enthusiastic keeper of his family’s documents, now maintained as the Sylvester Manor Archive at New York University. As a tribute to him, his wife, Alice (left), founded the Andrew Fiske Memorial Center for Archaeological Research at the University of Massachusetts Boston. She died in 2006.

Courtesy of Sylvester Manor

Eventually I will combine the results of a decade of research into Andy’s documents with information from oral and traditional histories; intellectual, economic, agricultural, and architectural history; and archaeology, dowsing, and dendrochronological analysis of the timbers of the house. The story of the house is restless, multicultural, and complex, reaching far beyond these grounds to the broader Atlantic world. Yet everyone who visits this marvel of a place (including me)—or who lives there—is momentarily seduced by the urge to leap over the gaps, to simplify, conjecture, or even make things up. The account Andy gives me contains much that is true, but I will learn it relies mostly on nineteenth-century family tales.

Andy’s highly-colored version starts like this: “In 1652, the dashing Nathaniel Sylvester, son of an English merchant family, sailed north from Barbados, where his family owned sugar plantations that depended on the labor of hundreds of Africans. With him came his teenage bride, Grizzell Brinley, daughter of Charles I’s court auditor, Thomas Brinley. After a dramatic shipwreck, newlyweds landed on Shelter Island, where a large and comfortable house (built previously by Nathaniel and his servants) stood ready to receive the young couple.”

Andy’s story continues to unroll with Nathaniel and Grizzell as “Quaker protectors,” rather than as the Quakers they undeniably were; a mention of the Quaker martyr, Mary Dyer, who fled the savage persecution of Boston Puritans to spend the last six months of her life on Shelter Island, before returning to Boston to be hanged; and a roll call of local Manhansett Indians, including the “sachem of all Long Island,” Youghco. Andy notes that generations of Sylvesters hobnobbed with, and married into, New England’s ruling elite, and that, last but not least, Grizzell bore twelve children, eleven of whom lived to maturity, thereby begetting the long line that led to himself.

Andy then fast-forwards two generations to Nathaniel’s grandson, the fashionable Brinley Sylvester, born in Newport in 1694, who inherited on his uncle’s death in 1704 but couldn’t take possession until 1735, after a long court case gave him ownership. To Brinley, the ramshackle eighty-year-old family house seemed very out of date; he and his servants tore it down and built this one. We are standing in the east parlor as Andy tells me this, a room of extraordinary beauty and strangeness. Beautiful because the proportions and the fully paneled walls are exquisite; strange, because there are only two coats of paint on these walls, through which the silvery old wood shows in patches. The bottom coat is that acid blue-green so fashionable in the mid-eighteenth century. The top coat is a modest biscuit color, emblematic of good taste, applied sometime in the 1840s, just before wall colors turned dark and Victorian.

Andy creaks open a door hidden by the patterned wallpaper of the front hall to reveal a walk-in safe, a vault that he has been told could withstand the hottest flames for six hours. The ceiling is about seven feet high, and the walls—lined with high dark chests and file cabinets—press in on us from every side. Old trunks crowd the red-tiled floor. Thousands of documents are housed here in these trunks, and many others in drawers, and in albums in which Andy has flattened some several hundred of his most precious ones.

We take the heaviest album into the dining room, where there is more light. On the mahogany table Andy quietly smooths out the original parchment charter for what had once been an 8,000-acre water-bounded domain—the entire island (the manor property is now 243 acres). Blobs of red wax, the personal stamps of the various parties, dot the wide sheet, from which dangles a large, handsome governmental seal. Signed in 1666 by Richard Nicolls, the first governor of the English colony of New York, the document defines the place, “by the Indyans formerly called by the name of Manhansucke Ahaquatzuwamocke and now commonly known as by the name of Shelter Island.”

In the west parlor, we gaze at a gorgeous nineteenth-century French panoramic wallpaper: snowy peaks, garden terrace balustrades topped with overflowing urns of flowers, and peacocks. I sign the guest book and turn to admire the mantel. To the left of it is a closed door.Walking toward it, Andy says “This leads to the slave staircase,” as if it were the most natural thing in the world to have a slave staircase in your early Georgian house on Long Island.

The staircase behind this door spirals up to the attic, Andy remarks, pointing out how narrow and high the steps are. But nobody walks up this pinched back staircase any longer: the steps are blocked by a collection of small vases and dishes, mostly crystal and metal, which glitter in the light from a glassed side entry. Later I can’t help but notice the luxurious treads of the front stairs, whose four-inch risers are so shallow I seem to float effortlessly upward.

The steep “slave staircase,” which begins in a hallway just outside a groundfloor parlor, spirals up from the second floor to the attic.

Andrew Bush

It begins to sink in: the “servants” who Andy said built the house were slaves. So were those who lived in the house built by Nathaniel and his wife in the 1650s. In common seventeenth- and eighteenth-century American parlance, slaves were often called servants, as were indentured servants, whose terms did not last for life and whose children would be free. As I listen to Andy’s glancing mention of the staircase, I realize I am face-to-face with slavery in the North.

In 1984, Andy is among the comparatively few who are aware of Northern slavery. His family lived with that knowledge as his ancestors lived with their captives. The door to memory was never closed. The unknown people who lived here and helped create this place step forward when Andy and I reach what he says some call the slave graveyard, others the Indian burying ground. As I drove in, I had noticed an old gray fence line, but missed a big boulder lying closer to the main drive. Now I see an inscription carved on the flat side: “Burying Ground of the Colored People of the Manor since 1651.” There are no gravestones in this graveyard under the pines. Andy tells me that more than 200 people may be buried here. I don’t ask where are the records, why are there no markers? I can guess the answer. These are the ones who slipped soundlessly through history.

By the time Andy’s house was built in the 1730s, Long Islanders owned more human chattel than any other group of colonists in the North. In outlying areas such as Shelter Island, up to half the workforce was enslaved. After the Revolution, in 1799, the new state enacted agonizingly complex graduated manumission laws. Buying and selling continued through the years until the last days of the “peculiar institution”: Sylvester Dering, Andy’s forebear and lord of the manor between 1785 and 1820, purchased a “Negro man Joseph” in 1810. In 1821, Dering’s widow emancipated London, the last of the slaves at the manor.

The house attic once provided quarters for slaves. The last enslaved African American at the manor was manumitted in 1821, only a few years before New York State law finally set all people free, on the Fourth of July, 1827.

Andrew Bush

A drawing of a schooner was scratched on the attic wall by Isaac Pharaoh (1824–ca. 1880), a Montaukett Indian who was indentured at the age of five to Samuel Gardiner. Gardiner and his wife, Mary (a great-great-granddaughter of Nathaniel Sylvester) acquired the manor in 1827. Although free to leave the manor at age twenty-one, Pharaoh spent his entire life there and was interred on the property, in the “Burying Ground of the Colored People of the Manor since 1651.”

Andrew Bush
New York State’s last slaves were set free on the Fourth of July, 1827, only thirty-four years before the outbreak of the Civil War. Because there was never a single cash crop nor a united slaveholding oligarchy to point a finger at, the ubiquitous presence of slavery, which shaped New York physically, financially, and socially in profound ways, was obliterated. After the Civil War, the legend of a victorious Abolitionist North expanded.

In 1996 I returned to do some research for an article about the manor garden. Andy had died of cardiac complications in 1992, at the age of eighty; Alice, in her late seventies, was searching for a way to memorialize him. Andy, who had loved his family papers so dearly, had also loved archaeology. Fortuitously, I had just hired Gresham O’Malley, a graduate student at the New York Botanical Garden, to make measured drawings as illustrations for my article, a narrow look at the manor’s Colonial Revival garden history. While Gresham and I tugged on strings and tape measures and yelled to each other across the hedges, he told me about his digger brother-in-law, Stephen A. Mrozowski of the University of Massachusetts Boston, who had excavated at Jamestown and other seventeenth-century sites in Virginia and New England. As Gresham talked over our sandwiches about his brother-in-law’s excavations, Alice realized this might be the way to keep Andy vividly close to her.

Later, I asked Gresham if he thought Steve would like to visit, maybe even consider Sylvester Manor as a project. On a drizzly winter day, Steve drove up with his wife and kids, absolutely prepared, he later told me, to say no, but as we walked toward the house, he bent down and, from among the pebbles on the back drive, picked up a piece of creamware pottery and then a piece of eighteenth-century blue and white porcelain. Then he met Alice, who with her characteristic mix of downhome style and ceremony offered Pringles in a silver bowl as just the right pre-luncheon hors d’oeuvre. That clinched the deal.

Field archaeology requires squads of people, wads of money, and a lab to process the “finds.” Alice soon funded a research center at UMass in memory of Andy. Steve came to the manor with a preliminary team in the summer of 1998, with the intention of unearthing what they called a multicultural Northern provisioning plantation that dates to the earliest days of European settlement on the Eastern Seaboard. Calling a farm in the North a “plantation” startled me, as did the concept that such places were specifically set up to provision West Indian sugar plantations powered by slaves. Steve’s digs would unearth the often voiceless “conversations” that had taken place on Shelter Island between Europeans, Indians, and Africans—struggles over power and the use of space revealed by artifacts and the faint, multilayered evidence of fences, roads, and buildings.

Once the UMass team started, the Sylvesters’ ease as travelers and merchants would also become visible as artifacts were teased out of the ground: English and European ceramics of every description, coins of fivenations, Dutch clay pipes and bricks, a German silver stickpin, and pounds and pounds of Caribbean coral, stockpiled as a vital ingredient for making mortar.

In the vault, sorting through stacks of early-twentieth-century letters, I open a small envelope without a name or address. It contains four half-size sheets covered in Cornelia Horsford’s loopy, acrobatic scrawl. Cornelia inherited the manor upon her mother’s death in 1903. Her letter (a copy she made) is dated Sylvester Manor, August 26, 1915, but has no salutation. I don’t expect to find more than family gossip, and indeed, the letter describes a visit to the manor by a cousin, General Sylvester Dering II. But on page three Cornelia writes, “I took him to see the old slave burying ground which grows more interesting day by day as new graves are brought to light from under the heavy brush, some mounds, some fallen in, all with their headstones and footstones.”

Headstones and footstones? Unbelievable. No headstones or footstones have ever been remarked on by anyone as far as I know.

I read Cornelia’s breathtaking sentence to a student archaeologist, Elizabeth Newman, who is helping me with the document inventory. The two of us drop everything and head outside through a sleety rain to investigate. We pound up the soggy, puddled drive to the graveyard under the pines and fumble our way through the gate, half expecting to find regular mounds and hollows, headstones and footstones, where we’ve never seen them before. Neither; nothing. Just brambles, withered field grasses, and poison ivy on the uneven ground. As we walk around, wondering what lies beneath our feet in this sacred space, we stare down at every step, feeling gullible, disconsolate. Elizabeth kicks a small stone like a kid. Then she says, “God, there are so many rocks in here, I guess because this area hasn’t been tilled, and the glacial rocks haven’t been cleared . . .”

Her voice trails off, and a light goes on.The two of us—she, an archaeologist, and I, a landscape historian—are used to scrutinizingthe earth for different patterns. We missed this one until Cornelia gave us the clue: we are stumbling over an army of small glacial boulders and fieldstone pebbles, many more, at a cursory glance, than anyplace else we have examined on the property. The largest have rolled or been rolled against the fence; the little ones are scattered underfoot, many half buried in the ice-glazed soil. Are these Cornelia’s “headstones and footstones”? Early Dutch settlers on Long Island used fieldstones as grave markers, sometimes carving them with names or initials and dates.

Cornelia Horsford (1861–1944), who inherited the estate in 1903, was mistress of the manor until her death. Photographed here on the occasion of the Baptism of the Calves, an annual tradition of her own devising, she stands in front of the manor’s iconic boxwoods.

Sylvester Manor Archive, New York University
As Jerome S. Handler and Ira Berlin have written about funeral practices in the Caribbean and the North American colonies, the slaves’ exclusion from segregated white cemeteries meant that their own graveyards became free spaces, repositories of African religious and cultural traditions. Resourceful, thoughtful people, slaves used funeral rites to renew traditional ceremonies surrounding death and to remember the chain of ancestors that connected them to their homelands. For the first Africans on Shelter Island and their American-born children, this ground would have meant something quite different from what it meant for the Sylvesters.

“The Burying Ground of the Colored People of the Manor since 1651” is so identified by an inscription on a boulder, carved about 1884. The site may have been an Indian burial ground before the arrival of European settlers.

The Fiske Center for Archaeological Research/University of Massachusetts Boston
In this cemetery under the pines, investigation has at last begun: in March 2013 UMass initiated geophysical testing, the most respectful and least invasive initial approach. Sixty to seventy burials have been tentatively identified in this first attempt. The site may have been an Indian burial sanctuary long before the Sylvesters and Africans arrived—Manhansetts and Montauketts reportedly lie there too. Although the manor team has been very careful to keep good relations with local tribal councils, the latter are understandably uneasy about allowing anyone to disturb the peace of the dead here: they have fresh memories of other Indian graveyards on the East End that were discovered in the course of construction and hastily covered up without ceremony. Further investigation will proceed with tribal and community assent. What can be discerned of body positions and funeral ornaments may clarify the ethnicity of any remains; signs of age and gender may emerge; the effects of labor and nutrition may be revealed in teeth and bones—although in the island’s acid soil, “remains” might only be stains in the sand.

Julia Dyd Havens Johnson (ca. 1809–1907): The manor’s housekeeper for many years, she was one of the few African Americans to own land on Shelter Island in the period following emancipation in New York State.

Sylvester Manor Archive, New York University
“Byron Griffing [Shelter Island’s town supervisor, 1892-1905] has the names, which I mean to have carved on a rough boulder on the crest of the slope at the head of Julia’s grave,” Cornelia’s letter continued. No stone confirms the presence of the remains of Julia Dyd Havens Johnson, the manor’s longtime housekeeper and the daughter of a former slave, who died in 1907. A search for Cornelia’s list of names and for Griffing’s records produce discouraging news: the farmhouse where his papers were stored was burned to the ground. The list, if it existed, is lost.

I’m angry at Cornelia’s careless exaggeration about proper headstones and footstones, and her inability to carry through with honoring Julia, someoneCornelia had known all her life. Given that people so often don’t do what they say, rage seems unreasonable. Yet both Cornelia’s professed good intention and her inaction put her gesture where it belongs: in the category of condescending and sentimental racism typical of her class and day.

If death, its rituals, and the dense shade of the pines have consecrated the “Burying Ground of the Colored People of the Manor,” everyday life, the seasonal rites of agriculture, and bright sunlight hallow the spot of earth that two nineteenth-century manor proprietors called the “Negro garden.” For their makers, both grave and garden carried the sweetness of order, an order that the slaves themselves created out of the disorder and powerlessness of their lives. In death, Africans believed, the spirits of those they buried would be free to return to Africa; every spring, the seeds they chose to plant on Shelter Island would sprout from the ground. Digging the first spadeful of earth—the same act performed in both places, life and death answering each other—restored harmony and gravitas to the world.

In a daybook entry for May 7, 1856, three years before his death in 1859, Samuel Smith Gardiner, the husband of Nathaniel and Grizzell’s great-greatgreat- granddaughter, Mary L’Hommedieu, noted matter-of-factly that he “Planted Six rows of corn in the Negro garden.” By then, twenty-nine years after the end of slavery in New York State, the term “Negro garden” had lost its original sense as the only ground that plantation slaves were permitted to cultivate for themselves. But when the first Sylvesters began to import Africans, probably through the West Indies, both masters and slaves would have known of such plots in Barbados. While traces and records of many “Negro gardens” can be found on plantations in the South, few besides this one have even been mentioned in the North, where plantations run with slave labor were fewer to start with, and where, once free African Americans decamped for the cities, the ground was used for other purposes.

Those gardens provided more than physical sustenance. The Barbadian planter Henry Drax (a friend of Nathaniel’s brother Constant) instructed his overseer, “For the enabling Negroes to go through their work with cheerfulness there must be great care taken that they have plantation provisions enough, besides the constant provision-ground of their own. The quantity and most convenient place for the Negroes Garden you must allot, which should be in the outskirts of the plantation.” Drax added that better food—particularly food they’d grown themselves—made his slaves better workers.

So where on Shelter Island was the “Negro garden”? Samuel Gardiner’s bare notation gave no clue. We had to depend on his less reassuringly literal son-in-law, Eben Norton Horsford. Writing thirty years after Gardiner’s daybook entry, he described the paths, tidal spring, shoreline indentations, and Indian village and graveyard of the manor grounds’ North Peninsula, finishing with a flourish: “Between the spring and the end of the graveyard, runs the old hawthorn hedge, the office of which has been evident as it forms the limit of the Negro Garden, immemorially so called, and indicated the distance to which the servants might extend their spading the ground.” (Horsford applied the word “servant,” as did both Nathaniel Sylvester and Andy Fiske, to enslaved as well as free laborers.) There are now a number of paths, springs, and silted-up marshy areas, but no visible signs of Manhansett burials or a hedge, as the entire peninsula has become a weedy succession forest.

The first tantalizing clue on the ground appeared when Steve and two graduate students, Anne Hancock and Lee Priddy, who would help lead the first summer of exploratory fieldwork, drove down from Boston on a cold March day in 1998. We walked together across the land bridge to the North Peninsula, which Steve, on the basis of the Gardiner and Horsford descriptions, had picked as a dig site for the season. Over the winter the ground had frozen, then thawed, heaving up stones and artifacts. Anne stooped down for the first find, a piece of a whiteand-red-glazed pot. Then, quite a few steps farther on, she picked up a sherd that fitted it exactly. Lee spotted something brownishly different from the brown leaves and soil. She turned it over in her hand, and then, in a star turn only an archaeologist can pull off, said, “It’s a piece of a clay colander, glazed inside, unglazed outside. The irregular holes tell me it dates to the seventeenth century.”

I held the fragment in my hand. For Steve, Anne, and Lee, as historical archaeologists, this was only a piece of a European utensil without a material frame of reference to tie it to a specific ethnic use or period of manor settlement. For me, this colander was a gathering basket. Garden produce is picked, washed, and sorted in a colander. Were we in the Africans’ garden? I felt the tension between the archaeologists’ thoughts and mine.

The UMass team opened units down along the south shore of the creek to seek evidence of the Negro garden. In the steamy summer days, shrieking with mosquitoes, graduate student Craig Cipolla led a team that cautiously cleared the thick brambles, bittersweet, saplings, and poison ivy and then even more cautiously lifted dense matted roots and topsoil. Faint traces of cultivation can be as powdery as moths’ wings—the marks of wooden plows or hoes are far less stable than those left by stone foundations. When Craig found long, shallow, crumbly scratches in the earth—“plow scars”—he called me away from the workroom to come see. They resembled the lines of a child’s secret letter written in lemon juice, whose invisible script appears only when the paper is held over heat—except here it was Craig telling me what I was looking at that made me see it.

The team went on to uncover possible planting pits or fence postholes and a larger pit containing ceramics dating to the late seventeenth and early eighteenth centuries, but not a trace of food waste—a good sign, since they were looking for a garden rather than a dwelling where people cooked and ate what they grew. A second season of excavation produced more inconclusive yet tantalizing evidence: a few more lines in the soil that were tentatively identified as plow scars, as well as finds ranging from pre-contact lithics (chipped stone artifacts) to antebellum ceramics from the era whenSamuel Gardiner planted his “six rows of corn in the Negro garden.”

Alice and I were looking at the portrait of Cornelia as a little girl that hangs at the bottom of the front stair. Cornelia’s eyes do not waver. They appear to be fixed on the gauzy middle distance so beloved of Victorian portrait painters. But Cornelia’s gaze is not gauzy—never mind that she is wearing a little pink frock with a frilled neckline, or that she is only about ten years old. If this girl had turned her head, she would have stared through Alice and me, taking us backward through history. As an adult and with nineteenth-century certitude, Cornelia had firmly reshaped the house and gardens to suit herself and her ideas of what the place had been.

Alice loved the portrait of Cornelia as a young girl. She loved all the family portraits in the house, but she felt especially close to Cornelia. “We divided a hundred years between us in this garden,” she said. It’s true: Cornelia started ordering garden seeds in 1890 and died in 1944. Alice came to the manor in 1952 and died in 2006.

With her demise, Steve Mrozowski decided that the first phase of the investigation—the University of Massachusetts’s six-week summer field schools (eight in all, from 1999 through 2006)—should conclude. The team moved their efforts to Boston, to the Andrew Fiske Memorial Center for Archaeological Research at the university. There they began the analysis (still ongoing as of 2013) of the thousands of artifacts from the digs. “Lab analysis takes ten times as long as fieldwork,” says Katherine F. Hayes, the field superintendent, so who knows when it will all be completed.

Cultivation of the windmill field: Run as an educational organic farm, the estate has returned to its agricultural roots. The windmill (now missing its sails), built in Southold, Long Island, around 1810 and subsequently relocated to Shelter Island, was installed at Sylvester Manor in 1926.

Courtesy of Sylvester Manor
One of Andy’s nephews, Eben Ostby, Eben Horsford’s namesake, inherited the manor. Eben, who has been at Pixar Animation Studios since the company’s start-up days in 1983 and is now the supervising technical director, lives in California and has no desire to recast himself as a Shelter Island squire. Fortunately, his sister’s son, Bennett Konesni, now thirty-one years old, a farmer and musician, was eager to take on the job. He moved into the manor house and set up an organic farm where more than 100 families (at last count) pick up weekly shares of vegetables. Fields fallow since the nineteenth century are now alive with young farm staff members and summer volunteers who learn—and teach—the arduous methods of organic farming. They plant garlic, nurse seedlings, stake tomatoes, make pickles and compost, tend poultry and porkers, and host hundreds of visitors to demonstrate that agriculture doesn’t have to be agribusiness.

Near the 1810 windmill and the recently fenced four acre windmill field, now a corduroy patchwork of crop rows, sits a new farm stand built with volunteer labor. A welcome addition to an island where in previous years almost the only vegetables to be found were canned or frozen, the stand offers hardy salad greens beginning in April and stays open well into winter, selling the potatoes and turnips and onions that have been cold-weather mainstays over the centuries for rich and poor, black and white.

Excerpted from The Manor: Three Centuries at a Slave Plantation on Long Island, by Mac Griswold; Farrar, Straus and Giroux.Copyright © 2013 by Mac Griswold.