I was four and dressed in shorts; the backs of my bare thighs were sticking to the pew. I knew that when the organ swelled and we all stood, my skin would pull away from the tacky varnish. That would be bad. I distracted myself with the hats. Which was the biggest? Which had the most feathers?

Just then a final couple came up the main aisle. They were well dressed like everyone, the man in a tan suit, the woman in a silk dress, gloves, hat. They slipped into the pew just in front of my family; and through the church behind us, people were suddenly murmuring. The couple was colored.

An usher came up the aisle, Mr. Brewster, my pal Tom’s father. He was a scoutmaster, and he was kind and funny. But now he looked very stern. He spoke to the colored people. "Folks," he said, "you can’t sit here. You’ll have to move to the back."

I saw the woman turn toward her husband. She was biting her lower lip. I saw the man’s shoulders go tense. "Why?" he said quietly. "Because," said Mr. Brewster, "that’s where you people sit."

The couple rose and stepped into the aisle. She clung tight to her scowling husband’s arm as they walked past us and down the aisle. I got on my knees on the bench and turned to watch. Mr. Brewster was standing like the angel outside Eden in the Bible history book, his arm extended, pointing at seats in the second-last pew. But the couple walked right past Mr. Brewster and out the church’s open doors. My jaw dropped. My brother, next to me, tugged my arm and whispered, "Sit back down!"

I did, and the lady on my other side smiled down at me. During the week she was behind the ice cream counter on West Street. She patted my knee.

"It’s all right, sonny," she said. "Those coloreds must be visiting from up north." She patted me again reassuringly. "They just didn’t know their place." Then the organ bellowed and everybody stood. I peeled my thighs away from the sticky varnish, dropped my feet to the floor. Standing, I couldn’t see much in the forest of adults, except straight ahead, where there were two empty seats.

I didn’t know the word "segregation," but I’d just experienced it.

That scene came back to me as Anne and I drove home last week from a wedding in Maryland. With Anne behind the wheel, I rode along, eyes closed, resurrecting the Annapolis of my boyhood. Not far below the Mason-Dixon line, in the very north of the South, it still was a very southern town. With the humid air off Chesapeake Bay, we kids breathed in segregation.

Twenty percent of the town was black; back then, the oldest were children of former slaves. Four thousand black people, and I knew only two of them.

One was Chapman, who came in horse and wagon to cut our grass and trim shrubs, and who always called us boys, "Mr. Denny" and "Mr. Jim." (We idolized him, but still addressed him by his last name; just "Chapman.") The other was his wife Miss Lou, an immense, jolly woman who was housebound from a stroke. My mother had taken me along when she dropped off bundles of clothes at their tiny house outside town.

Miss Lou lived in an overstuffed chair in their dark parlor, watching the world through the brightness of the open front door. She was funny and warm-hearted and so full of love that I could feel it. Once, as Ma drove us home, I upset her by saying I wished Miss Lou was my aunt. "Don’t say such things," she said. I asked why. "They’re different people from us," she said, eyes on the road ahead.

The 4,000 colored people lived in our midst, though in their own neighborhoods. They had their own food and clothing stores, movie theater, and restaurants; their own schools; their own barbershops and beauty parlors, their own doctors and undertakers.

And their own churches, too; though, as far as I could tell, they worshiped our God. The black professionals took their families to St. Phillip’s Episcopal, and the rest attended Mount Moriah African Methodist or Gethsemane Baptist or storefront churhes set among their stores. A few dozen ventured down Duke of Gloucester Street to attend St. Mary’s Catholic Church. And sit in the back.

I’ll tell you more about that place and those times next week; but let me share, to my shame, an incident when I was about four. It shows how early we breathe in what sociologists call "generally received opinions."

My mother had got my big brother, 10, to take me along as he walked to the ice cream store with his pal Charley Steele.

We’d walked out of our neighborhood and onto West Street when we passed a small black boy sitting on a stoop. My age, my size, he grinned at me and held out a stick of gum. But when I reached for it, he snatched it back, laughing.

Instantly, I was flooded with rage. I shouted, made a fist, and bopped him on top of the head. He began to cry, and bigger kids came out of the house.

Denny and Charley grabbed me by the arms. They rushed me down an alley and back into the white neighborhood. Then Denny, ashen, knelt and shook me. "Don’t ever do that again!" he yelled. "You could get us killed!" Then they hauled me home, my feet barely touching the pavement.

I felt ashamed that I’d caused trouble to the big boys. Now, recalling that day, it’s a deeper shame: at what I’d shouted fiercely at that other little kid.

I’d shouted, "Don’t you dare pull a trick on a white boy!"

Read about Jim Atwell’s new book, "From Fly Creek _ Celebrating Life in Leatherstocking Country" at JimAtwell.com.