I could find no developer commentary explaining why the spells needed to drop a prismatic sphere's or prismatic wall's layers were picked for any edition of Dungeons and Dragons that preceded Pathfinder, and Dungeons and Dragons 3.5e is whence Pathfinder takes its prismatic sphere, spray, and wall.
The prismatic spray originates with Vance…
Vance's short story "Mazirian the Magician" (1950) is the prismatic spray spell's source:
Mazirian made a selection from his books and with great effort forced five spells upon his brain: Phandaal's Gyrator, Felojun's Second Hypnotic Spell, The Excellent Prismatic Spray, The Charm of Untiring Nourishment, and the Spell of the Omnipotent Sphere. This accomplished, Mazirian drank wine and retired to his couch.…
[Much later] Mazirian shook off the [foe's] spell, if such it were, and uttered a spell of his own, and all the valley was lit by streaming darts of fire, lashing in from all directions to split Thrang's blundering body in a thousand places. This was the Excellent Prismatic Spray—many-colored stabbing lines. Thrang was dead almost at once, purple blood flowing from countless holes where the radiant rain had pierced him.
As can be seen from the description, Vance's the Excellent Prismatic Spray spell doesn't much resemble the effects of the D&D spell prismatic spray (from which Pathfinder drew its spell of the same name). Further, I've found nothing to indicate that the prismatic spray spell's effect was changed to distance it from Vance's creation, or, instead, that the spell was named as an homage to Vance's invention, or, really, anything much about the spell's development at all. But it's an almost inconceivable coincidence that the name's accidental, given Gygax's affection for Vance's works.
…But the prismatic wall and prismatic sphere don't seem to
While in the same story Mazirian uses the Spell of the Omnipotent Sphere, the D&D spell prismatic sphere doesn't seem to owe Vance's spell anything:
He called his charm, the Spell of the Omnipotent Sphere. A film of force formed around his body, expanding to push aside all that resisted. When the marble ruins had been thrust back, he destroyed the sphere, regained his feet, and glared about for the woman.
To this reader, that sounds closer to (but still different from) Otiluke's resilient sphere or something.
Thus it's my understanding and the understanding of others far more well-informed than I that the spells prismatic sphere and prismatic wall were created created wholecloth for Dungeons and Dragons, with no fiction serving as inspiration. Some things are just new, I guess.
Research notes
Originally—that is, in the Player's Handbook (1977) for Advanced Dungeons and Dragons, which is as originally as I can muster—the necessary spells were, in order of layers dropped, cone of cold, gust of wind, disintegrate, passwall, magic missile, continual light, and dispel magic.
However, in order of spell level for magic-user—AD&D's wizard—, these are as follows: 1st—magic missile; 2nd—continual light; 3rd—dispel magic, gust of wind; 5th—cone of cold, passwall; and 6th—disintegrate. This comes really close to a magic-user needing to devote one spell per spell levels 1st through 6th to drop a prismatic wall or sphere. (The last layer's being dropped by dispel magic remains an outlier but a reasonable one.) It'd be great if I could find a version of passwall or cone of cold that was, instead, a 4th-level spell, but, alas, I can't. (Even the 1974 Dungeons and Dragons puts its spell pass-wall as a 5th-level spell and lacks the spell cone of cold entirely!)
Were the spells needed to bring down a prismatic sphere or wall to equal one spell per spell level, this would be a predictable and interesting design choice, but there appears to be no point in mandating two 5th-level spells and one 6th-level spell: A level 10 magic-user can cast two 5th-level spells but no 6th-level spells, yet a level 12 magic-user can cast four 5th-level spells and one 6th-level spell, more than enough to drop a prismatic sphere or wall and have spells remaining!
Thinking this might be campaign-dependent, I looked for AD&D liches. While liches are (thankfully!) rare, encounters with level 17 magic-users in AD&D are rarer, and I thought perhaps an early adventure module might be geared toward level 12 or higher PCs encountering foes who could cast prismatic sphere or wall. But, for example, Asberdies, the infamous lich from Descent into the Depths of the Earth (1978) that's for levels 9–14, doesn't memorize prismatic sphere or wall, nor do the two liches (and, incidentally, the ki-rin) from the Rogue's Gallery (1980) (which includes PCs from Gygax's campaign). The spells needed to drop prismatic layers appear oddly—but, perhaps, appropriately—random. (I mean, seriously, what magic-user memorizes continual light instead of stinking cloud?)
In Dungeons and Dragons, Third Edition makes some changes, but keeps the same weird non-pattern. The spells, in order of layers dropped, become cone of cold, gust of wind, disintegrate, passwall, magic missile, daylight, and dispel magic. While some names changed, spell levels didn't, so that, in order of wizard spell level, this is as follows: 1st—magic missile; 2nd—daylight; 3rd—gust of wind; dispel magic; 5th—cone of cold, passwall; 6th—disintegrate. The Dungeons and Dragons 3.5 revision swaps the spell levels of daylight and gust of wind, keeping intact the same non-pattern.
Spell level isn't the connection. Spell school isn't the connection (a spell from each school would've been just as convenient!). Expected character level seems to have no impact. It doesn't appear to be a puzzle, either, as there's no reasonable acronym or initialism or apparent code. Some stuff we may just never know unless a designer deigns to reveal his secrets.
Yes, area effects will cross through the walls
There is no language in any of the layers, or the general wall itself, that state it acts as cover or that spell effects can't pass through it.
Prismatic Wall is described as(emphasis mine):
A shimmering, multicolored plane of light forms a vertical opaque wall--up to 90 feet long, 30 feet high, and 1 inch thick--centered on a point you can see within range.
The wall itself is only light, with each layer providing a specific effect. An area of effect spell, such as fireball is not physically obstructed from passing through, so it can pass through.
Casting Through vs Effects passing through
The only relevant wall type is violet, which states:
While this layer is in place, spells can't be cast through the wall.
The limitation here is being able to cast through the wall. In this case, the spell isn't being cast through it. If the point of origin is outside the wall, then you are not casting through the wall. Your point of origin is outside, so you can cast. The spell effects can pass through because there is no limitation on that, only casting through it. The language from globe of invulnerability covers this specific case - and because the violet wall doesn't have that language, then creatures behind the wall can still be affected by a spell that is cast on the other side.
The creatures under the effects of the spell just happened to be within range of your point of origin. If your point, and therefore your spellcasting, doesn't cross the violet wall, then targets within range of that point but on the other side of the wall would still be effected.
A spell is only being cast through if the spellcasting portion of it crosses. For Area of Effect spells, like fireball, that is not happening. The effects of the spell cross over, but the spell is not being cast through the barrier.
Best Answer
Firstly, it's worth mentioning that the wording "or a similar spell of equal or higher level" and "or another spell of equal or greater level" is likely future-proofing to a certain degree, in case they bring out something like (or someone homebrews something like) "greater dispel magic" or something in the future, then if this specifically said "This layer is destroyed by dispel magic." then RAW, greater dispel magic wouldn't work, even though it clearly should.
Nonetheless, some of these layers have existing spells that would bring down the layers in question besides using the examples given. Let's tackle them one by one:
Orange Layer
This layer "is destroyed by a strong wind." A strong wind? Well, firstly, I'll include Joel Harmon's comment:
Unfortunately, as the comment says, there are no "hard number"; this appears to be describing an environmental effect, so yes, if someone cast prismatic wall outside on a very windy day, a DM could rule that this layer might go down immediately, but in the middle of the dungeon, that's not going to happen. So we need to use magic.
The gust cantrip isn't going to cut it; gust of wind, wind wall, warding wind and whirlwind, on the other hand, all create strong enough wind to take this wall down.
From gust of wind (PHB, pg. 248):
From wind wall (PHB, pg. 288):
From warding wind (XGtE, pg. 170):
From whirlwind (XGtE, pg. 171):
Ok, this last one doesn't contain the phrase "strong wind", but it's a whirlwind! I think it counts...
Green Layer
This layer requires "A passwall spell, or another spell of equal or greater level that can open a portal on a solid surface". Unfortunately, the only other portal related spell I can think of that uses a surface is teleportation circle, which specifies "the ground" (PHB, pg. 282):
I imagine the intent of this wall's requirement is to cast the spell similar to passwall on the wall itself, not on the floor next to it, so I don't think teleportation circle can help us.
There is demiplane (thanks @Nick), which says (PHB, p. 231):
I think this would count as creating a spell of greater level that can open a portal on a solid surface, so demiplane should bypass the green layer.
(Thanks to @illustro for suggesting the next three spells)
Mordenkainen's magnificent mansion could be an option, since it somewhat ambigiously says (PHB, pg. 261):
Gate says a similar thing (PHB, pg. 244):
These entrances or portals could be on the green wall (gate less obviously, but it could be oriented to be against the wall), thus serving as an alternative to passwall, although they're not explicitly cast against the wall like passwall is, so it's a DM's call as to whether this would work.
Arcane gate could also be an option, given that you could align the free-floating portals with the wall (PHB, pg. 214):
The fact that it's hovering, it could be argued that this is therefore definitely not interacting with the wall, but the mentioning of rotation could imply that it could be aligned with the wall, as with gate.
Another suggestion (thanks to @CapnZapp) could be using the Nolzur's Marvelous Pigments (DMG, pg. 183), a very rare magic item that can:
I, as DM, would personally allow this as it seems similar enough to passwall for the purposes of taking down this layer, although it must be said that RAW this magic item does not cast passwall, nor does it cast "another spell of equal or greater level", so strictly RAW this one probably wouldn't work.
Overall, many of these options would be up to the DM, since passwall and demiplane seem to be the only spells that explicitly say that it's cast on a solid surface.
Indigo Layer
This layer requires "bright light shed by a daylight spell or a similar spell of equal or higher level." Those that spring to mind (or @illustro's mind) are dawn, wall of light, sunbeam and sunburst, all of which create bright light, although I should mention that clearly the light cantrip will not work (even though that's obvious) since it's not of equal or higher level to daylight.
From dawn (XGtE, pg. 153-154):
From wall of light (XGtE, pg. 170):
From sunbeam (PHB, pg. 279):
From sunburst (PHB, pg. 279):
Also, if we include magic items, it's worth mentioning that the Driftglobe can cast the daylight spell, and that there's a magic item called the Gem of Brightness, which says (DMG, pg. 171):
Unfortunately, this won't help us since it wants to target a creature, and we need it to take down a wall. However, the next bullet point says:
The only trouble with the Gem of Brightness is that whether the effects of an uncommon magic item are considered "of equal or higher level" to a daylight spell, and the fact that the description of this layer of the prismatic wall specifically says "or a similar spell", and the effect of a magic item is not a spell (although a DM may allow it anyway, since it's light from a magical source with a similar effect to daylight; I'd allow it).
Driftglobe is probably alright RAW because it mentions the daylight spell specifically.
Violet Layer
"This layer is destroyed by a dispel magic spell or a similar spell of equal or higher level that can end spells and magical effects." Besides dispel magic, what else can "end spells or magic effects" in such a way?
Possibly greater resortation, since it can remove certain conditions like petrified and the like, but I'm not sure if it is similar enough to dispel magic, since it just stop a creature being affected by such spells or effects. This one might have to be a DM's call as well.
On the other hand, remove curse should be valid RAW, as it is considered to end curse spells; from bestow curse (PHB, pg. 218):
Hex also says this about it (PHB, pg. 251):
So remove curse is called out as being able to end spells early at least twice, meeting the criteria for being a spell "that can end spells and magical effects".
It's also worth pointing out that an antimagic field wouldn't work, since prismatic wall specifically says (PHB, pg. 269):