As volatility in the stock market grows, a handful of experts are raising an alarm about the rise of index ETFs and mutual funds, which has never accounted for this much of the market before.

They warn that the unprecedented amount of index ETFs trading in the market — index ETFs accounted for nearly 30 percent of the trading in the U.S. equities market last summer — could magnify, or even cause, flash crashes.

In turn, that may put individual investors, who are increasingly invested in index funds, more at risk. And many may not realize how exposed they are to the risks of a relatively small group of stocks held in the major indexes, said experts.

Tim McCarthy, a former president of San Francisco-based Charles Schwab and Japan's Nikko Asset Management, has been a longtime proponent of index investing. But he now advises that investors diversify their investment styles as well as their asset classes.

He suggested investors move 25 percent to 50 percent of their equity portfolios into actively managed absolute return funds, preferably those with a 10-year track record and a relatively small amount of assets of between $1 billion to $2 billion. (Research has shown over the years that active managers stand their best chance of success before their assets under management grows too high.)

As always, he said, investors should look for low fees.

He said he has grown increasingly uneasy about the risks based on the hypergrowth of index funds, and the price difference between stocks outside and inside index funds.

From 2007 through 2014, index domestic equity mutual funds and ETFs received $1 trillion in net new cash and reinvested dividends, according to the Washington, D.C.-based Investment Company Institute. In contrast, actively managed domestic equity mutual funds experienced a net outflow of $659 billion, including reinvested dividends, from 2007 to 2014.

Meanwhile, the price of the underlying equities in index funds is rising, though no one is sure exactly why. Research by S&P Capital IQ, as of Dec. 31, found stocks that were in the Russell 2000 were trading at a 50 percent premium to stocks that were not, up from 12 percent in 2006. The statistics are based on median price-to-book ratio.

That kind of price difference is seen by some as a kind of canary in the coal mine, indicating that there is a bubble in the stocks of companies held in index funds — and that their prices could come down further and faster than other stocks in a downturn. In turn, that could put pressure on the share prices of the index mutual funds and ETFs themselves.

"It's complicated, but it could be a very big problem," said David Pope, managing director of quantamental research at S&P Capital IQ. He and colleague Frank Zhao studied the liquidity in the market for the S&P 500 last summer and identified the 10 stocks that had the biggest difference in liquidity at that time, compared with the index. They included ExxonMobil, Berkshire Hathaway, Johnson & Johnson,Microsoft, General Electric, Wells Fargo, Procter & Gamble, JPMorgan Chase, Pfizer and PepsiCo.

"A structural problem may arise when the liquidity demanded by the ETF exceeds the liquidity availability of some of the underlying holdings," they wrote.

Basically, if an investor wants to sell an index fund as the market declines, the managers of the fund might have trouble selling some of the stocks in the fund. An active manager could choose to sell any stock in her fund and thus potentially navigate a downturn better. But an index fund manager has to sell exactly the shares held in the index in the same proportion as demanded by the index. If the fund manager doesn't find a buyer for, say, shares of ExxonMobil, the price of ExxonMobil will fall until a buyer is found.

While market theorists have always recognized this as a potential problem for index investing, no one has been sure exactly how it would play out or when problems might arise. As long as there are enough buyers and sellers actively setting prices and trading, index funds and stocks should pose no extra risk. It's just that no one is sure exactly how many is enough.

Indeed, not everyone thinks McCarthy is right, and others point to different risks as bigger causes for concern, including the unknown impact of the way that high-frequency traders place orders.

"So we have two new factors when it comes to a potential market situation," said John Rekenthaler, vice president of research for Chicago-based Morningstar. "There are always new factors. Most of the time, new factors don't play out according to expectations."

He pointed out that two decades ago, people worried about what the impact of 401(k)s would be in the market and whether non-professional investors would be apt to sell more quickly in a downturn. The opposite turned out to be the case.

Even if the risk posed by index investing is rising, the growth in index funds doesn't necessarily pose a huge system risk, pointed out Sean Collins, senior director of industry and financial analysis for the Washington, D.C.-based Investment Company Institute. "The share of assets going into index funds is rising. Does that necessarily cause markets to be dysfunctional? The answer is no," he said.

He pointed out how much more diversity there is now in index investing. Much of the money flowing into index funds has been going into markets in which there hasn't been much indexing before, including emerging markets equities and bond markets.

McCarthy said investors would be wise to look at their portfolios with the emerging risk of index funds in mind. There's not much an individual investor can do to guard against the risks posed by high-frequency trading, short of bowing out of the market entirely.

But there are some steps investors could take to manage the risks posed by an index fund-dominated market.

In addition to investing some of their stock portfolios in actively managed funds, McCarthy suggests investors take a hard look at how diversified they are.

First, he said, an investor could make sure he or she isn't double-exposed to the same stocks. He cited the case of a friend of his, a doctor, who had invested in blue-chip stocks, some mutual funds and in an S&P 500 fund that turned out to hold — guess what — many of the same blue chips and tech stocks. In the downturn in 2000–2001, he lost 50 percent of his portfolio.

Every market is different, McCarthy said. But in part because of the flow of money into index funds, the U.S. equities market has become more dominated by a handful of big technology stocks. That's something that index fund investors, like his doctor friend, may not easily recognize now.

As someone who has managed the back end of trading systems, McCarthy said he is increasingly uneasy about the level of index investing and has begun to give speeches about the potential dangers of a market in which a growing number of managers are hamstrung by the requirement that they match their indexes.

But he knows that he's at the leading edge of people talking about it — and that many think he is warning too hard and too fast. "This is unfamiliar territory for me," he acknowledged. "But index investing has so much power, and it's derivative-priced.

"Sometimes it's better to be vaguely right than exactly wrong," he said.

— By Elizabeth MacBride, special to CNBC.com