Some people with disabilities feel they're being asked to stay home so that 'everybody else can get on with life'