Dr. Sammy Lee, the first Asian-American man to win an Olympic gold medal, died over the weekend after battling pneumonia. He was 96.

In the 1930s, Southern California had enough of the South in it that young Sammy Lee could only watch through the iron fence most days when other boys his age swam at the pool in Pasadena's Brookside Park. The pool, like the area's beaches and many other public facilities, was segregated. But not on Wednesdays.

The park declared that Wednesdays were International Days. "Basically, anyone who wasn't white could use the pool," said Paula K. Yoo, one of Lee's biographers. "Then they'd drain it afterward."

Young Sammy, though, had developed a passion for diving, and was determined to practice more often than once a week. In was to become his typical response to prejudice, he found another way.

"He's very much a problem-solver," Yoo said. "So when he was told, 'You can't use the pool except for that day,' he decided, 'Okay, then I'm going to work with a coach who would help me.' "

Lee found a coach and they worked on his diving over a sand pit, which Yoo said was not uncommon back then. And sand had one advantage over water: "It gave him stronger leg muscles, which is why he was able to jump so high and perform those beautifully executed triple-somersault dives," she said. " Ironically, racism made him a better diver."

The extra strength made Lee a good enough diver that he decided the Olympic team would be his goal. But he had another challenge, one at least as formidable as racial prejudice: his father.

Lee was born in the U.S. to immigrant parents who'd made tremendous sacrifices for their child: He was their American dream. His father, Soonkee Rhee (he changed his name after arriving in America), wanted Sammy to be a doctor. He thought his son should be spending as much time on his studies as he did at diving practice. When Lee went to Occidental College, he did a lot of studying and a lot of diving.

"He'd spend a lot of time in labs, then walk 800 feet to the pool house, and practice dives to unwind," Yoo said. "Then (he'd) go back to the lab and work some more."

Lee was good enough to make the Olympic team, but the Games were canceled twice because of World War II. He finally got his chance in 1948. He was a 28-year-old medical corpsman in the Army Reserves competing in the London Olympics. As a hushed audience watched, Lee ascended to the high platform, jumped, and performed a triple somersault before elegantly slicing through the water. When he swam to the surface, he discovered he had near-perfect scores.

He'd compete once more in Helsinki four years later during the Korean War. As Major Sammy Lee, he almost didn't go: Lee thought he needed to tend to the troops. The Army thought otherwise, gave him a month to train, and urged him to go. Lee won his second gold, making him the only Asian-American to win two consecutive Olympic gold medals. (He also earned a bronze in London for the 3-meter springboard.)

As a civilian, Lee discovered that his status as a veteran didn't shield him from prejudice. He and his wife Rosalind were turned away when they wanted to buy a home in one part of Orange County. Eventually, they bought a home nearby from a sympathetic developer. Eventually they owned a house with a pool, where Lee coached students. He also coached divers for the 1960 Rome Olympics. Later, he'd mentor Olympic gold medalist Greg Louganis, and he served as an ambassador to the Olympics under three presidents.

Copyright 2016 NPR. To see more, visit http://www.npr.org/.