You need to know how to calculate the required CFM - cubic feet per minute - that your ductwork will allow. On a room per room basis you need to determine how much air flow is needed, how large the vent needs to be, and thus how large of a duct you require. Then you need to determine how large of a trunk you require and at what points you need to reduce it after branching off legs from the trunk.
Finally you need to also know where to position the return air and how large it needs to be, in order to support the output from your HVAC.
I don't know how to calculate those things - I just know that's what you need to know.
Given that this question is 3 years old you probably moved on... But I can make a couple of non-expert suggestions...
The first thing I would look at is shade. Can you add some foil into the windows to cut down the heat in the morning? Or some shade cloth on the outside? I have large sliding doors facing west and in the summer I span some 50% black shade cloth from the eave to the ground about 6' out. One can see through quite fine and it cuts down the heat load tremendously (even with the low-e windows I have). This saves a lot of money too...
WRT your specific question, here is what I would do. I you want to go to a ceiling register, check out https://www.hvacquick.com/products/residential/Grilles-Registers/Curved-Blade-Grilles/TRUaire-A303-Series-Triple-Deflection-Curved-Blade-Grilles
Click on the links tab and then on the performance data link. You get a table that shows grille size on the left and then performance data across the row keyed by CFM. Take a 10x10 grille, assume you have 200CFM, that means the air "jet" will reach 12ft horizontally ("throw"), and the noise is "NC 30", which is low (there's no lower entry in that table, but other types of grilles/manufacturers go down to NC 20).
Now the other two interesting measurements are at the top of the table, for that register @200CFM you have a "back pressure" of 0.022 "WC and 600fpm face velocity. You probably have something like https://www.hvacquick.com/products/residential/Grilles-Registers/Grilles/TRUaire-210-Series-Steel-Single-Deflection-Adjustable-Bar-Grilles now, look at the performance info for the 10x6 you have. At 200CFM you have 15' throw (interpolating a bit), 0.026 "WC pressure loss, and 650fpm face velocity. So you would be reducing the pressure by a rather small amount, so you wouldn't get much more air.
Now, if you have an an anemometer to measure wind you could work your way backwards by measuring face velocity. I got a https://www.amazon.com/dp/B01L9SC36A but cheaper will work too. So measure the fpm you have now by holding the anemometer in front of the center of the grille, use the table for the grille you have now to get CFM, then use that to look up the hypothetical new grille to see throw distance and noise level. Note that for ceiling you want curved blades so the air mixes at ceiling level and down't blow straight down on you. You could also go with a round ceiling grille...
If the new grille has lower pressure loss, you will get more CFM, stealing from other outlets. WRT duct, 8" flex doing 200 CFM has a friction loss of 0.024 "WC (http://www.hartandcooley.com/tools/friction-loss-calculator-for-flexible-ducts). Of course I don't know how long your duct is, but it doesn't look like it's going to be a limiting factor, but it may well be in the same range as the grille.
OK, I'm not an expert and an expert would tell you you have to recalculate and rebalance the whole system from scratch if you make a change. And the expert would be right. But you asked how to "wing it, and it doesn't have to be perfect" and I hope that the above gives you a way to make some back-of-the-envelope calculations so you end up in the right ballpark.
Best Answer
Such a connection introduces more friction to the system, reducing the available flow. If the system is sized adequately to accommodate this, then yes, it is OK. If I understand you correctly, this large round duct you are connecting to the air handler is serving as an extended plenum. Partly depending on the relative size of the duct and air handler, this may be the one place where it would be worth constructing a proper transition fitting instead of a simple flat adapter.
In a way, the flat adapter is crippling the entire system before it even gets a chance to get going, just my opinion, based on my innate desire for efficiency. As stated earlier, if the system can accommodate an inefficient transition, then it is OK to use it. I just happen to not like it.