Have you ever wondered why we love sad songs, or get choked up at a “Thank you, Mom” Olympics commercial? Questions like these were the impetus for Susan Cain’s new book, “Bittersweet: How Sorrow and Longing Make Us Whole.”

“Bittersweetness is the hidden source of our moonshots, masterpieces and love stories,” writes Ms. Cain, who believes that we experience our deepest states of love, happiness, awe and creativity precisely because life is imperfect, not in spite of that fact. At the heart of her exploration is the naming and reframing of her titular paradox: that there is no bitter without sweet.

“Bittersweet,” which is part memoir and partly a look at neuroscience, psychology, spirituality, religion, epigenetics, music, poetry and art, makes a case for the underappreciated “curiously piercing joy at the beauty of the world” within a culture of relentless optimism. The book aims to explain that irrepressible lump in our throat spurred by seeing an image of our high school grad as a grinning toddler.

“The sadness from which compassion springs is a pro-social emotion, an agent of connection and love,” she writes. And this “happiness of melancholy” has a physiological signature and explanation.

It turns out, Ms. Cain writes, that the vagus nerve — the constellation of nerves that connects the brain stem to the throat and the abdomen and is responsible for digestion, breathing and heart rate — is also associated with compassion in the face of sadness, our instinct to protect our young and desire to experience pleasure.

Fittingly, the oldest, most instinctive part of our nervous system, which evolved so that we had the necessary empathy to respond to our underdeveloped newborns, Ms. Cain indicates, is also the site of the very sadness-joy-survival continuum that makes us human.

Ms. Cain, who is also the author of “Quiet: The Power of Introverts in a World That Can’t Stop Talking,” discussed the importance of sorrow and more in the edited interview below.

What would you like people to understand about being open to or celebrating feelings like sadness and longing?

S. C.: We would do better to understand that the most fundamental aspect of being human is the longing to live in a more perfect and beautiful world than the one that we live in now. Sometimes that’s expressed in explicitly religious terms, like the longing for Mecca or for Zion, or for Eden, or like the way the Sufis put it, which is my favorite, “the longing for the beloved of the soul.”

But it’s also in those moments when we see a gorgeous waterfall or a painting that’s so beautiful that it makes us cry. That’s a spiritual impulse that we’re having. What we’re really seeing is an expression of that more perfect and beautiful world that we feel like we come from and that we need to return to.

Tell us a little about the importance of “longing,” how it’s been misunderstood in modern times and within the context of a culture driven by “the tyranny of optimism?”

S.C.: In our culture, you say the word “longing” and you might think “mired in longing” or “wallowing in longing,” but that’s not how it has been understood historically. In the “Odyssey,” Odysseus was seized by homesickness and that was what propelled him on his journey.

That’s what carries you to the divine, to creativity. I don’t believe we should be making a distinction between the divine and creativity and compassion and all these things. They’re all manifestations of the same fundamental state of humanity.

If you had published this book before the pandemic, do you think there would be a different level of reception?

S.C.: When I gave my TED Talk on bittersweetness in the summer of 2019, it was fascinating how much the very act of talking about sorrow, longing and bittersweetness was seen as being a statement of depression, as opposed to a cleareyed view of what life is.

The fact that all humans have to go through that together is one of our deepest sources of communion and one of our deepest sources of art and beauty. I think it was very hard for half the audience to grasp that at that moment in time. I think if I were giving that talk today, it might be different.

You make a big distinction between sweet melancholy and depression. How do you define the difference?

S.C.: I am melancholic by nature, but I think of myself as a happy melancholic. I’m actually not depressive in the clinical sense of that term.

It’s really interesting because there’s a long tradition that goes back centuries of talking about melancholy and its mysterious virtues — more than 2,000 years ago Aristotle was asking why it is that many of the great poets and philosophers and politicians have a melancholic personality. Melancholy and depression are two distinct states, but often no distinction is made.

What fields of psychology are bucking this tendency toward pathologizing melancholy?

S.C.: One psychologist, Dacher Keltner, who I wrote about in the book, has done pioneering work on what he calls the “compassionate instinct,” and he points out that the very word “compassion” means suffering together. So what you’re doing when you’re feeling compassionate is actually experiencing this sorrow of others.

When we think of human nature, we often either cynically or despairingly go to the idea of survival of the fittest, but Dr. Keltner says we should also really be talking about survival of the kindest, because as humans, the only way that we survive is by being able to respond to the cries of our infants. What has radiated outward from there is that we’re not only responding to our own infants’ cries, we react to the cries of other people’s infants and then we react to other human beings in distress in general.

Can listening to bittersweet, minor-key music, prime you for “the bittersweet mind-set” and life’s fragility?

S.C.: Yes, absolutely. In fact, that was actually the catalyst that got me to start writing this book. I would listen to a technically sad song, but what it made me feel instead was a sense of communion with other people who had also known the sorrow that the music was expressing. And with this incredible sense of awe and gratitude toward the musician for being able to translate what had clearly originated in pain and to transform it into beauty. It’s kind of like my church when I listen to that music. My playlist is on Spotify, actually.

What are your “bittersweet” practices?

S.C.: Meditation is something that I practice on and off, along with mindfulness. But I’m also really interested in exploring any experiences that make me feel more connected to a state of love. There’s another practice that I’ve started doing over the last year or so that came out of the pandemic.

During the beginning of the pandemic, I fell into this habit of doomscrolling Twitter. It was what I would do first thing in the morning as I woke up. I decided that was really unhealthy. I was thinking of the poem by Rumi where he talks about how we wake up every morning, empty and frightened, and instead of going straight to our study, we should pull down the musical instrument and let the beauty be what you do.

So I decided to start my mornings instead with beauty. I asked on Twitter for people to recommend their favorite art accounts and I started following them. And now my feed is full of art. Before I do anything else, I take the time to pair the art with a favorite poem or an idea that I’m thinking about or whatever. It’s a daily practice that I love.