What makes creme brulee set

In this case, the mixing order won't make that much of a difference. You need the sugar dissolved and the yolks broken up.
However, adding the sugar to the yolks will have two results:
- it will allow you to more easily break up the yolks but if you mix too much it could actually aerate them- not a big risk but not good thing for creme brulee.
- more importantly the sugar will give you a little bit of buffer to keep the yolks from curdling when you mix in the hot cream.

As for using just yolks vs yolks and whites.

Yolks have a lot of protein but they are very high in fat. Whites are just blobs of protein, of course. Using only yolks creates custards that are very rich and creamy but also high in fat (which is where "rich and creamy" comes from). Using the whites gives some of the needed binding power while reducing the fat a little.
Additionally- whole eggs are easier to use because the eggs don't have to be separated and you have fewer extra whites to use elsewhere.

Removing a custard (which is what creme brulee) is from its form or mold does not definitely require agar agar. Flan, which is famous from a number of cuisines, is an unmolded custard.

While experimentation would be required, it is highly likely that if you use a silicone based flexible form, and make a fairly stiff custard, you will be able to gently and carefully unmold it. You may wish to lightly grease the form before baking the custard to help facilitate release. This Chest of Books article recommends two eggs per cup if milk for a custard that requires unmolding.

Many molded custard recipes exist, although a good number of them have gelatin as an additional stabilizing ingredient. You may wish to google "molded custard recipe" or "Charlotte recipe" for inspiration (Charlottes tend to have a lady finger garnish around the edges).

If you are interested in using agar agar or other modernist hydro-colloids in your recipe development, you may wish to read the excellent primer at Cooking Issues.

Note that they characterize the texture of agar agar gels as somewhat brittle, which may not be ideal for your application.

See also Can I substitute agar agar for gelatin in pudding which is a very similar application. Michael at Herbivoracious indicates that agar agar may not be suitable due to the texture. There are some alternate ideas there that may help you get started.