For two decades, academic researchers have asked the same questions about black males in science, technology, engineering, and mathematics, known as the STEM fields:

Why do black males underperform in grade-school and high-school math and science classrooms?

Why do so few pursue STEM degrees?

Of those who enter college with the intention to major in STEM fields, why do so many switch to other disciplines?

And among those who persist and graduate with science majors, why do so few proceed to Ph.D. programs?

The scarcity of black men earning STEM degrees has been documented repeatedly. For example, among American citizens and permanent residents, the proportion of black men at the Ph.D. level more than doubled between 1992 and 2012, but from a very low base of only 1 percent to 2 percent of all STEM degrees, according to the National Science Foundation’s annual "Survey of Earned Doctorates." (These figures exclude psychology and the social sciences.) In 1992, black men earned only 139 of 11,485 STEM doctorates awarded, and in 2012, 334 of 16,545 STEM doctorates.

In establishing why progress has been so slow, there is no single answer, says Earnestine P. Easter, a program director in the division of graduate education at the foundation. Black males face more than a few obstacles before they reach college: white teachers who misinterpret their behavior, high suspension rates, school closures in urban neighborhoods, inadequate schools, concerns about "acting white," single-parent households, poverty, violence, and a lack of positive images of successful black males.

Another reason is that the matter doesn’t receive sustained, consistent attention. "We know there are some policy drivers in our country, and when there are alerts that something is threatening U.S. competitiveness and economic development, we tend to respond so that agencies and private foundations can intervene," Ms. Easter says. "For black males, we have signals and longitudinal reports starting from preschool all the way through high school indicating problems with black males not achieving, and that means there’s a much smaller pool of STEM students at the undergrad level."

Some scholars say that while the research and data collected are important, the fixation on these racial disparities contributes to a pervasive narrative that reinforces broader stereotypes of black men as an endangered, lazy, and incompetent group in crisis.

"Anyone who takes time to read about them could confidently conclude that black male students are troubled, their future is bleak, they all do poorly, and there is little that can be done to reverse longstanding outcomes and disparities that render them the least likely to succeed," says Shaun R. Harper, executive director of the Center for the Study of Race and Equity in Education at the University of Pennsylvania.

"I’m not saying that we abandon the data and explanatory undercurrents of these questions," he says. "But statistics help maintain notions of white supremacy in that they very powerfully reinforce that white folks are, and very much belong, on top because people of color just can’t seem to get their act together."

Ansley Abraham, director of the Southern Regional Education Board’s State Doctoral Scholars Program, echoes those sentiments. "You know what they say about data—it can be twisted and used in many ways," he says. "However, the end result of these data on black men is that the system is not working for this group of people. So we have to ask what’s wrong with the system if it consistently works for certain people and not for everyone."

Mr. Harper and other researchers have documented a movement, going back at least 15 years, by many well-intentioned people who have called attention to the issues faced by black male students in STEM fields. Among the factors are academic and cultural isolation, the difficulty of performing in the face of negative stereotypes and low expectations among faculty members, a lack of mentors of color and friendship networks, concerns about financial debt, inadequate advising and emotional support during times of stress, and lack of exposure to hands-on research.

Some scholars have also argued, in reports and academic journals over the years, that the movement to broaden minority participation has tended to focus more on "fixing" the black male student than on addressing the structural and institutional forces that undermine his academic achievement and sense of belonging on campus.

A growing number of colleges have created black-male-oriented institutes, centers, summits, mentoring programs, and campus initiatives. Since 1998, the National Science Foundation has allocated more than $385-million in research funds to understand barriers to African-Americans in STEM fields, in addition to strategies to help them succeed. Some of the work is being done by faculty members at historically black colleges, which graduate large numbers of black students who go on to earn doctorates in science and engineering.

Howard University is using NSF money for a project "to disentangle the issues of race and gender and their relationship to factors that influence STEM interest and success" among black males. Faculty members at Washington State University are studying the career pathways of black male college students who are pursuing IT-related careers. Vanderbilt University is looking at minority-mentoring initiatives for black doctoral students and postdoctoral researchers in engineering, while the University of Central Florida identifies best practices and lessons that can be applied within STEM graduate programs to broaden participation and increase success rates.

NSF officials report progress in serving underrepresented groups in the agency’s competitive-grants programs. From 2007 to 2014, 270 black males have been awarded the foundation’s graduate research fellowships, with the largest increases over the past five years. A few black males have received support from the agency’s early-career-development program.

Despite significant federal investments in STEM education for black males, the rate of increase in their enrollment remains sluggish compared with those of other groups. Low completion rates in postsecondary-degree programs are most pronounced among black males.

The news is both good and disturbing. "The raw numbers of black men earning Ph.D.’s have doubled," says Mr. Abraham, of the doctoral-scholars program, which gives individual awards of $20,000 to Ph.D. scholars annually, 35 percent of them in STEM fields. "But we’re not making up any ground. We’ve had two decades worth of affirmative action and diversity efforts, and we’re not even holding steady. That is disturbing. It should be disturbing to us all."

A recent report by the American Institutes for Research notes that women (1 in 5) and blacks (1 in 5) are most likely to leave science careers, academic or otherwise. The study found that 21 percent of blacks—compared with 17 percent of whites, 14 percent of Asians, and 14 percent of Hispanics, leave STEM fields, with 42 percent of black men opting to work in government.

"The STEM community is losing contributions to scientific and technological discovery," said Lori Turk-Bicakci, the report’s lead author and senior researcher. "This brain drain restricts potential advantages gained from diverse perspectives and the ability of role models for underrepresented groups."

What’s more, employment rates for new Ph.D.’s in STEM fields have been down or stagnant in all of those major disciplines for the past 20 years, according to an analysis published by Slate.

Student-loan debt is another problem. Another recent report by the research institutes found that HBCUs, led by Howard University, were key producers of black STEM Ph.D.’s from 2005 to 2010. (Howard awarded one-third of all STEM doctorates conferred by HBCUs during that period, the report said.) Yet these graduates received less financial aid than their black peers at predominantly white institutions: 19 percent of black graduates of predominantly white institutions graduated owing more than $30,000, compared with 30 percent who received degrees from historically black institutions, the report said.

"Minorities are risk-averse," says Ms. Easter, of the National Science Foundation. "They don’t like a whole lot of debt, and that’s why they don’t go on to Ph.D. programs in larger numbers."

Given the attention and money devoted to attracting more black males and improving their participation in STEM fields, why have 15 years of efforts not yielded better results? And what should well-intentioned educators and policy makers keep in mind as they undertake new efforts?

Some agree that it’s essential to stop fixating on negative data and start telling the stories of black success.

"The conversation is usually about the 2 percent," says Mr. Harper, the education scholar at Penn. "It’s problematic, for sure. But there are 334 black men who earned STEM Ph.D.’s in 2012, and we never hear about their success and the enablers of their success."

Mr. Harper, who is black, says he earned his Ph.D. 12 years ago and can count on one hand the number of people who stopped and asked him about his journey to success. "It’s easier to download data and lean on statistics," he says. "It takes time to seek out successful blacks and spend time doing a turn-by-turn deeply textured analysis of their trajectory."

Black students are often apprehensive about going into STEM programs after hearing about the low success rates and hostile environments that students face, says Ms. Easter.

"But there are black Ph.D.’s who have made it in spite of these obstacles," she says. "Those who have managed to survive need to tell their stories. We have a lot of quantitative data, but what we don’t have is rich qualitative data that can serve as a guide or case study for those who want to understand and manage their own professional growth and development in a culture and environment that’s not really supportive."

In the edited conversations that follow, four black men who have earned STEM Ph.D.’s tell their stories. They share their different journeys, which reflect the dynamics, challenges, and—most important, they believe—the value of investing in the lives and education of black men.