Dr. Vivek Murthy said many people are struggling, and there are consequences.
The event was described as a town hall on community, connection and health with the U.S. surgeon general.
He said his mission is to help lay the foundation for a healthier country relying on the best scientific information.
The key, Dr. Murthy said, is human connection, "It feels nice to be together with people in person. I miss this a lot,"
The surgeon general addressed an audience of about 200 at Manny's Cafe in the Mission neighborhood.
He described the isolation and loneliness made worse by the pandemic as a social recession and called for building a social revival.
"How do we rebuild connections in our lives and rebuild communities across America. That's the most urgent public health issue that we're dealing with," Dr. Murthy said the social isolation during COVID affects not only older people but spans all ages.
Audience members said they came to see what they can do to help themselves and others .
"COVID-19 was hard for all of us. I was living alone during the pandemic. That's got a personal touch to me as well," said Humphrey Obuobi of San Francisco.
"There's a huge gap right now between the communities we used to rely on and communities we're left with. I'm hoping to find ways to bring people back together," said Logan Dirkx of San Francisco.
The surgeon general said there's a shame that comes with loneliness and that it can cause physical health problems,
"How do we break that cycle? The more shame you feel, the less worthy you feel. It makes it harder to interact and engage with other people which is why loneliness is a downward spiral."
Dr. Murthy said the changes need to come from individuals and institutions such as workplaces and schools.
He said value should be placed on fulfilling, meaningful relationships over wealth, power and fame, "How do we prioritize people, spend time with the people we love. Make decisions about where we put our time, energy and attention based on the people we care about in our lives."
The surgeon general said workplaces can design environments that promote and support human connection.
And that schools should help children build a strong foundation for social and emotional learning.