Can I downgrade few user licenses instead of downgrading the entire subscription?

Yes, you can downgrade your user licenses at any point during your subscription if you have more than 3 user licenses.