Its been a long time since I did any fluids, and even then, it was not much. But I'll take a shot just from simple logic.
Suppose we have a single fan at the front end of the duct. It is rated for 650 CFM, and without any duct there, you will get essentially that. But suppose you put this fan in a duct. Moving air will see friction against the walls of the duct, slowing down the air flow. The back pressure reduces the flow, so you really don't get a full 650 CFM out the end of the duct. Of course, bends in the duct hurt even more, but you said it was a straight duct.
Now, suppose you add a second fan at the tail end of the duct. This is sucking air out at a nominal 650 CFM rate too. But in the end, all it does is give you something closer to the full 650 rating for the two fans. You don't get 1300 CFM, you get roughly 650 CFM.
With two fans in parallel, both feeding into the duck (or both pulling air out) if they are not limited by air flow through a duct that is big enough to handle one fan, then you might get more air through in theory. But the problem is, now you have higher velocity air. The link that gregmac provided tells us that pressure losses are proportional to the square of velocity. If you could double the velocity (and in order to double the air flow, you must double the velocity of the air as it moves through the duct) then you would quadruple those frictional losses.
It gets worse. As is pointed out in gregmac's link, with two fans right next to each other, the air flow into those fans will interfere with each other. So you really wont achieve the full rating for those fans if they are set side by side.
How about the 800 CFM fans? If they are too big for the size duct you have, then you again will be limited by the back pressure. You may end up with only a net of 650 CFM anyway.
It seems to make most sense to have a second duct. Run two ducts in parallel, both adequately sized for the fans they are fed by. Yes, I know this may not be an option.
Basically you are trying to balance your new ducts with respect to your old ducts by sizing the new ducts. If you make a mistake or if your relative cooling loads (office vs rest of the house) ever changes, your only recourse would be to replace the ducts.
My preference is to size the ducts as large as reasonable possible (without increasing cost too much) and use a damper to tune the airflow ratios. Since you have 12" duct and a place to attach them, just get a damper like this:
http://www.grainger.com/Grainger/DAYTON-Round-Balancing-Damper-2TFY1?Pid=search
An uninsulated unshaded west wall in Texas can probably generate a very high cooling load in the afternoon.
Best Answer
At the first level you need to compare cross sectional area of the 18" round duct with the cross sectional area of the four 8" oval ducts. There is a secondary effect of using the multiple smaller ducts because it exposes the air flow to more inside surface area which increases the pressure drop some but that can be compensated for by making the replacement duct area be larger than the original.
The cross sectional area of an 18" diameter duct is:
I do not know the shape of the 8" oval duct that you were considering but assuming that it is a standard ellipse shape like this:
We can see that the A dimension would be half of the 8" size and the B dimension would be half of the standard 2x4 wall stud width of 3.5"
If using four of the oval ducts the total area would be:
21.99 * 4 = 87.96 sq inches.
So to compare 254.45 from the large duct to the 87.96 from the four smaller ones you can see that only about a third of the required area is provided. So the answer would be no, this is not a suitable replacement.
Of course if the 8" oval duct has a slightly different shape this could affect the calculation some but I'd still say that it wouldn't be enough.
Here is a graphical comparison:
Update for Obround Ducting
The obround ducting available from one source is 3" by 8" in size (assuming the 8" specified by the original poster).
The cross sectional area of a duct this size:
So for the 3x8 size being considered it would take (254.45 / 22.06) = 11.53 ducts to be equivalent to the one 18" round duct!!!