Point House Future Hangs in Balance as Owner, Village Squabble

[caption id="attachment_41861" align="alignnone" width="647"] The Point House in North Haven. Stephen J. Kotz photo[/caption]

By Stephen J. Kotz

The owner of North Haven’s historic Point House this week said if the village doesn’t like the way he plans to renovate it, it should be willing to pick up the tab for the difference. If not, he said he was prepared to let the colonial-era house — possibly the oldest in the village — deteriorate to the point where it would have to be razed.

“I have tried to be a good neighbor since the day I moved in,” said Stuart Hersch, the president and chief executive officer of Cantor Fitzgerald, who bought the 2.6-acre waterfront property in 2013 for $10 million from the model Christie Brinkley. “When I bought the house I agreed with the mayor to move it to another part of the property. I didn’t agree to spend whatever he and the village tells me to.”

[caption id="attachment_57832" align="alignright" width="432"] An entryway to the Point House.[/caption]

The house, believed to have been built in 1804, if not earlier, according to a report prepared by historic consultant Zach Studenroth for the village Architectural Review Board last summer, once occupied a prominent perch overlooking Sag Harbor. When Mr. Hersch bought the property at the corner of Ferry and Fahys roads, he wanted to replace the Point House with a new home and first sought to have it moved from the property or even taken apart and sold in pieces to raise money for charity.

But that set off alarms in the village, which has no historic district, and Mayor Jeff Sander brokered a deal in which Mr. Hersch would apply to the village Zoning Board of Appeals for a variance to have two houses on a single lot. Once that variance was approved in May 2014, a new foundation was poured and the house moved to a new location fronting on Ferry Road. There it has sat ever since, its interior gutted, some of its windows dangling open to the elements.

Last month, Mayor Sander, speaking at a village board meeting, expressed his concern that the house, once thought to be saved, was in jeopardy. Mr. Hersch, he said, wanted the village to give him tax breaks and other funds to help preserve the house that was built by John Payne Jr., a prominent merchant, and later lived in by members of the Fahys family, whose watchcase factory played a dominant role in Sag Harbor’s economy.

Mr. Hersch, who said he did not want to be painted as a villain, said he would be happy to renovate the house, but only to his specifications. He has already poured about $200,000 into the project, he said.

“I was fortunate enough to make some money,” he said. “But I believe we live in America and I will decide how to spend my money.”

Mr. Hersch said he grew up in a two-bedroom apartment in Brighton Beach in Brooklyn, where he and his brother shared a bedroom with his grandparents. A lawyer by training, he said he received his lucky break when the owners of King World Television asked him to run the business. It later reeled off hit television shows like “Jeopardy,” “Wheel of Fortune,” and “Oprah.”

He was semi-retired, he said, when Cantor Fitzgerald, headquartered in the World Trade Center, was decimated by the September 11 terrorist attacks, and he was asked to join the firm to help it rebuild.

[caption id="attachment_57831" align="alignleft" width="461"] A plaque with details about the Point House hangs on the outside of the building.[/caption]

Mr. Hersch said he hired contractors “to clean up” the interior of the house, which he said was not habitable when he bought it, before it was moved. He said he understood it was his responsibility to restore the interior but balked at what he said were the village’s unfair demands for the exterior work.

This week, Mr. Sander said he did not understand how Mr. Hersch could be complaining about the cost of renovating the house when he has yet to submit any plans to the village to do so.

“We would like him to commit to repair the house in a manner that is consistent with what the ARB has requested,” Mr. Sander said, “and prevent it from deteriorating further.”

In his report, Mr. Studenroth noted that with the framing now exposed, it can be inferred that the house was built earlier than the federal period of 1790 to 1820, although he did not offer an estimate of when it may have been constructed.

As part of a restoration plan, he recommended that window sashes and their glass be preserved. If Mr. Hersch wants to install energy-efficient windows, they should be mounted on the interior, his report stated. It also calls for the removal of skylights and repairs to the siding and roofing shingles with historically accurate cedar. It also called for the restoration of the front door to a historic condition.

[caption id="attachment_57833" align="alignright" width="473"] The interior of the house has fallen into a state of disrepair.[/caption]

Mr. Hersch said his contractor had informed him the cost of those repairs would be “substantial,” but would not elaborate about what he would be willing to do.

Mr. Sander said Southampton Town has already steeply reduced the assessment on the house, saving Mr. Hersch thousands annually in real estate taxes and that the town had also offered a façade easement that would have paid him $175,000, which he rejected as too restrictive.

“Here’s a guy who paid $10 million for the property and probably $3 million plus for a new house and he won’t spend 300 bucks to fix a window?” asked Mr. Sander.

While acknowledging the purchase price was “a lot of dough for a kid from Brooklyn,” Mr. Hersch said how much he spent on the property was nobody’s business and insisted he wanted to be reasonable. He said he had no personal use for the house — his new home has plenty of space for guests, he said.

Still, he said he would try to preserve it, even allow it to be used for a community benefit, such as a home for an artist-in-residence at Bay Street Theater. That idea had been informally floated by him but not fleshed out, he said. But he would only do so, he said, if did not cost him anything, adding he was capable of making his own charitable donations.

One thing is for certain. Mr. Hersch said he is willing to tear down the house. “They can’t bully me,” he said of the village. “If they persist, it is going to deteriorate and my lawyers have said when it becomes a hazard it should be taken down.”