It was an unseasonably warm day in October, nearly five years after the Parrish opened the doors of its new Herzog & DeMeuron-designed building on Montauk Highway in Water Mill.

A half decade on, Terrie Sultan, the museum’s director, was reflecting on where the institution has come in that time.

“This is the first year we’ve had the tall grasses, which everyone assured me would come,” Ms. Sultan said as she gazed out through the wall of windows lining her office at the Parrish Art Museum. “They said that when these grasses come in, they’ll be 4 feet tall. Now, I’m starting to see it.”

Indeed, the expansive, meadow-like views tell the tale and are replete not only with grasses but colorful wildflowers and trees in their fall finery.

But the literal landscape is just one way in which the Parrish Art Museum has come into its own since opening day—November 12, 2012. Inside, there are hundreds of other ways to chart the substantial growth.

One hundred installations, 550 public programs, 421 new acquisitions, 330,000 people who have engaged with the museum through exhibitions, programs or events. These are just some of the numbers associated with the past five years in the life of the Parrish.

“When we designed this building, we wanted to make sure we do a perfect job of realizing the core values of this museum as reflected in the program going forward,” Ms. Sultan said. “The architecture is absolutely stunning—my colleagues who come to this building proclaim the beauty and correctness of it.

“But that’s only part of it. The building has to be the envelope for what happens inside.”

One of the major ways the Parrish has grown by the numbers is in terms of exhibition capabilities. With more than 12,000 square feet of exhibition space, curators have been able to offer thematic shows that were never possible in the museum’s original location on Jobs Lane in Southampton Village.

In celebration of the Parrish’s fifth anniversary, this weekend the museum will host a full slate of programming to captivate lovers of both art and architecture—music, art exhibitions, gallery talks and a cocktail party will all be part of the festivities.

“It’ll be a great weekend,” Ms. Sultan said, “and I think it will be similar to opening weekend five years ago.”

The celebration kicks off on Friday, November 10, at 6 p.m. with “The Impact of Architecture,” a discussion between Ms. Sultan and Cathleen McGuigan, editor-in-chief of Architectural Record, about the significance of the Herzog & DeMeuron-designed building—the first museum the firm has built on the East Coast.

“We’ll be talking about architecture in the service of a society. It’s really about how buildings function as social engagement loci,” she said. “One of the reasons the Parrish trustees realized we needed to expand was we didn’t have an opportunity to display our collection in a way that an organization with 3,000 objects should.

“Special exhibits are great, but it’s the ongoing engagement with the objects that belong to this community that’s really important.”

One of the key aspects of the building has been the way in which the galleries allow for juxtaposition in highlighting work from the permanent collection with that of contemporary artists. Curators are able to zero in on specific creative themes various artists may have in common, or a current artist may be able to reflect on the legacy and imagery of an important artistic figure from the East End’s past.

“Five and Forward,” a new exhibition opening on Saturday, is a prime example of this use, as it features nearly 100 works by artists who represent the East End’s history as well as its continuing artistic legacy. The show will remain on view at the Parrish for the next year and it encompasses seven galleries, three of which will focus on specific artists—including 19th century painter William Merritt Chase.

“Since we opened, most of our Chases have been out in the world, other museums have borrowed them,” Ms. Sultan explained. “The reunion of our Chases will be in one gallery. Another gallery will be dedicated to Alan Shields and a third to James Brooks in celebration of our new arrangement.”

That arrangement is with the James and Charlotte Brooks Foundation, which has given the Parrish 89 works by James Brooks and his wife, Charlotte Park. It also established a fund to support the collection and present exhibitions of the artists’ work, and is an example of the type of gift the Parrish is now in a position to accept and showcase.

In addition to Chase, Brooks and Shields, “Five and Forward,” also highlights work from the collection selected by guest curator Rashid Johnson. On Saturday at the member’s reception, East End artists Max Blagg, Michael Combs, Eric Dever, Bill Komoski, Bastienne Schmidt, Ned Smyth and others will speak briefly on a specific gallery or work of art in “Five and Forward” that is of particular meaning to them.

This strong engagement with current artists reflecting on works from the past is the type of community interaction that Ms. Sultan has found to be a particularly pleasant aspect of the last five years.

“We always knew this building would be a home for art, and that it should be a center for cultural engagement with the community,” Ms. Sultan said. “But now we understand what that purpose means. It makes my heart soar when I see how the community is coming here and using the building, whether there are 400 or 1,400 people in the building. We thought we understood that—then we saw it.”

Ms. Sultan recalled that preceding the opening of the new museum in 2012, the staff had been on site for two years while it was under construction. Though Ms. Sultan and her staff long imagined what opening day would look like, the reality of that day was something else entirely.

“The moment we opened the doors to the public, and 1,000 people walked in and started doing what they were supposed to do, I can’t tell you how uplifting that was,” she said, smiling at the memory. “They went into the galleries. They were laughing, or talking about the paintings, seeing the performances or just experiencing the art and the architecture.

“The planning of the program is philosophical and intangible until the minute it starts having meaning to people looking at it,” she said. “The building is a beautiful artwork, but it’s an empty vessel until the people get inside. That’s why I still go into the galleries two or three times a day to experience what others are experiencing.”

Another pleasant surprise is the way in which the Parrish Art Museum has become not only a center of the artistic community, but a place where the community itself can come, experience and actually shape the programming the museum offers.

“The opening of the building was the beginning, not the end,” Ms. Sultan said. “It’s now about seeing how much you can do and still maintain the same level of intimacy. That’s what we want to guard here. We want to be successful and we want everyone to visit … but not all at once.”