Paul Giamatti is stepping back into Downton Abbey—and this time, he’s doing it with a mix of reverence, surprise, and a deep sense of honoring a legendary co-star.

As he prepares for the release of his upcoming Black Mirror episode, Giamatti is also opening up about his return to the iconic period drama in Downton Abbey: The Grand Finale, a film that carries the weight of saying goodbye to Dame Maggie Smith.

“It was not the same without her, but it felt very much like in her honor in some way. She certainly still feels present through the whole thing. Not having her there is obviously a huge loss, but everybody was still acting in her spirit,” Giamatti told PEOPLE, reflecting on Smith’s passing and the emotional tone her absence brought to the set.

The actor, who originally popped into the Downton universe during the Season 4 Christmas special as Harold Levinson—Cora’s (Elizabeth McGovern) spirited American brother—admitted that rejoining the cast after all these years wasn’t something he ever expected. 

In fact, he was just as surprised as everyone else.

“The interesting thing for me was that they even asked me at all because I was like, ‘I don’t have an important character in this story.’ But what’s funny is my character has a very significant role. He does something very significant in the movie,” he teased, clearly holding back spoilers with a wink and a nudge.

While Giamatti didn’t get much screentime with Maggie Smith during his original run, the moments they did share left an impression. 

“Working with her the first time around was one of the best things ever and I didn’t even really get to do that much with her,” he recalled.

The final chapter of Downton won’t be short on familiar faces. Alongside returning cast members like McGovern, Michelle Dockery, Jim Carter, Phyllis Logan, and Laura Carmichael, The Grand Finale also welcomes fresh energy with newcomers Joely Richardson, Alessandro Nivola, Simon Russell Beale, and Arty Froushan. 

And with Giamatti’s Harold now playing a more pivotal role than before, it looks like fans can expect a finale filled with both heart and a few surprises.