How to make vassal from your own provinces

It sounds like your coalition partners opted out of allowing the coalition leader to negotiate for them. Which would result in what you describe. Not being able to cede fellow coalition members provinces etc.

This bit me when I did the opposite and forgot to untick the 'leader can negotiate for us' tick box. I got pulled into a war by Scotland when I was Ireland and Scotland ceded Meath to England in a peace deal before I could get my armies in a position to help.

The following is a fairly good description of coalition mechanics when at war.

Taken from the EU4 wiki -

Coalitions in war

• The coalition call-to-arms is an automatic one and will result in all members with or without a truce with the target and not currently at war with another coalition member instantly being part of the war. There is no option to decline.
• Each coalition has a war leader. If the coalition is attacking, the war leader is the country that declared war. If the coalition is defending, the war leader is the most powerful nation in the coalition (determined how?)
• The war leader always negotiates for the entire coalition; unlike alliances, countries in a coalition cannot make a separate peace. This means that if countries opt out of allowing the leader of a coalition to negotiate for them they will not be a way to gain provinces from them in that specific war.
• Occupying all of the provinces of a member of a coalition will allow a country to ask for that member as a vassal in the peace deal.
• War leaders may call non-coalition allies into the war, but these allies are not bound by the coalition rules - they may decline the call, and can sign a separate peace with the enemy.

I don't remember this happening before patch 1.2, but now it's certainly possible. You can demand provinces for your allies, and they pay the diplomacy cost themselves, where appropriate. The warscore cost is the same as usual.

Incidentally, returning cores to allies/vassals now gives a significant relations boost: +40 per core.