Staffing at the HUD office that pays for housing and support services across the country is slated to be cut by 84%.