Jan 13, 2015 2:26 PM

If Dan Gasby closes his eyes, he can remember it clearly.

He was standing on the beach next to his wife, restaurateur and lifestyle ace Barbara “B.” Smith, in Bay Head, New Jersey. It was the mid-1980s, and a woman was running down to them.

She was furious.

“Why are you here?” she demanded of the African-American couple. “I know you don’t belong.”

It is a moment Mr. Gasby will never forget—being hated simply for the color of his skin. He has carried it with him everywhere since, from visits back to his childhood home in Brooklyn, and out to the East End.

Until he discovered Sag Harbor. In particular, Azurest, Ninevah and Sag Harbor Hills—three distinct African-American enclaves connected by a long stretch of beach that were once a refuge from racism, and some say still are.

“Back then, it was the first time I ever saw an African-American in a summer environment, with a summer home,” Mr. Gasby said on Monday in a telephone interview. “And looking gorgeous and barbecuing, and not in an inner-city environment, not in a park listening to music. These were well-educated people who were achievers.

“As did Barbara, separately, I fell in love with the whole notion of being in a place where you could let your hair down and you weren’t judged,” he continued. “And you weren’t going to be treated with disrespect.”

Though lesser known than the big three communities, the historic Eastville neighborhood dates back even further, to the earliest whaling days in Sag Harbor. One of the oldest Native American and African-American settlements on Long Island, it was settled by whaleboat crews—the majority of whom were Africans, Shinnecock and Montaukett Indians—and runaway slaves. The area thrived until the economy took a downturn in the early 1900s, forcing the residents to leave for Brooklyn and Manhattan in search of work.

But word of the beachfront destination spread, and soon black vacationers sought out Eastville, ushering in the start of African-American second-home owners on the South Fork. By the middle of the century, the first streets of Azurest were drawn up, shortly followed by Sag Harbor Hill and Ninevah. It was a typical resort town—except nearly everyone was black.

What the multi-ethnic community has grown into today, along with its changing demographics and real estate values, is the focus of the Oprah Winfrey Network’s newest one-hour special, “Sag Harbor,” which will debut on Sunday, January 25.

“We’re calling it a love story,” Sheri Salata, co-president of Oprah Winfrey Network, said on Monday during a telephone interview from Harpo Studios in Chicago. “And one of the best kinds, because it stands the test of time. [‘Sag Harbor’] takes you on a journey you might not ordinarily get to go on, unless you know somebody.”

With a “very reasonable” budget, Ms. Salata said, the OWN crew of no more than 10 infiltrated the Sag Harbor community starting in July, first casting for the special—from sommeliers and real estate agents to celebrities and longtime summer vacationers—and then shot for just a few days over the course of three weeks, leading up to Labor Day.

“When I was listening to the idea, I was thinking, ‘Wow, this sounds like something our audience would eat with a spoon, to really understand a socially and culturally relevant community that has such beautiful, historic roots,’” Ms. Salata said. “We did a development meeting with Oprah, and it was the thing that day she most sparked to, for sure.”

But in recent years, the community is slowly moving away from its roots as predominantly African-American, especially as vacationers discover the beaches, Mr. Gasby said. It is no longer solely an escape from racism and inner-city life. And some residents are sad to see that magic, and mystery, disappear.

“The reality is, the world is moving forward,” Mr. Gasby said, “and either you move with it, or it moves without you. Nothing stays static. Nothing stays.”

The hour-long special “Sag Harbor” will make its world premiere on Sunday, January 25, at 10 p.m. on OWN. For more information, visit oprah.com/own-own/Welcome-to-Sag-Harbor.