Housing starts fell 5.5 percent in May to a seasonally adjusted 1.09 million annual rate, the NAHB said, citing government data.

Multifamily project permits declined 9.7 percent to a seasonally adjusted annual rate of 289,000 units. Single-family permits dropped 3.9 percent to 794,000.

"Today's report is consistent with builder sentiment in the housing market, indicating some weakness after a strong start to the year," said Granger MacDonald, chairman of the National Association of Home Builders and a home builder and developer from Kerrville, Texas. "Ongoing job growth, rising demand and low mortgage rates should keep the single-family sector moving forward this year, even as builders deal with ongoing shortages of lots and labor."

Robert Dietz, the association’s chief economist, said the lower numbers were not a surprise.

"After a strong start for single-family building this year, recent months have recorded softer readings," he said. "However, on a year-to-date basis, single-family starts are up 7.2 percent as builders add inventory to the market."