Why is it that 400w tubular metal halide lamps can only be operated enclosed, while standard metal halide lamps can be operated open so long as they are vertical?

Metal halide lamps sometimes explodes at EOL, shattering the outer jacket and presents a safety hazard. To contain the fragments of the arctube inside the outer jacket, a quartz shroud surrounding the arctube, preventing the arctube fragments to shatter the outer jackets.
Tubular MH lamps are usually used in area floodlighting and street lighting, an application which expects the use of enclosed fitting anyway, so no shroud present around the arctube. Standard elliptical and BT MH lamps, are usually used in high bay and low bay fixtures, which are often open, so they have a shroud around the arctube.

There are elliptical lamps without shroud, which are approved for use in open luminaires in vertical burning position and in enclosed luminaires in any burning position

To me this means that in such lamps, the design is not about how to absorb the energy of exploding arc tube parts, but about making the arc tube sturdy enough to not tend to explode in vertical burning position (which does not tend to bulge the arc tube, creating the defect that causes the explosion)

Lamps of all shapes with and without starting probe have been made, so the shape of the lamp does not directly imply the type of luminaire it is made for

I'm aware only about MV's and maybe early MHs (before the explosion was considered, only UV exposure was anticipated as a risk), with a disconnect operating on broken outer (either a piece of filament fuse burning out in presence of an oxygen, or a mechanical contact)...
But very early it was clear preventing the arctube from exploding is just impossible.
Some makers were experimenting with an external plastic containment (a silicone or similar sleeve designed to contain all the remains of the outer), but these designs disappear (extra cost offering just the open rating, nothing else).
Some makers were claiming if the lamp was shut off at least every xx hours, the arctube will rupture during warmup, when there is large stress in the quartz, but not yet much pressure. These were then restricted for vertical only, to avoid tensile stress in the outer (the outer holding the lamp weight sideways), so to help it to retain sufficient strength to contain the arctube explosion. But because it cost nothing in production, it was rather popular, even for such questionable method (safety relying on how it is operated; I don't think such approach would "fly" today)

I think it was with the pulse start lamps, when the quartz shrouds became the primary choice to reach open rating. The reason was, without the starting probe, the shroudwas not that bigh and (as its side effect) slowed down Na reacting with the quartz, so prolonging the lamp life and improving the color stability, so there were other benefits added on top of the open rating to justify the shroud cost...

Because this happens.....
Affectionately known as ‘lamp cancer’

I re-lamped/re-ballasted over fifty 400W PS MH BU parking lots at night a year averaging over 60 fixtures per store and only saw very few EOL exploded lamps in those years. Same way with the HOR lamps. We did get good tans though.  

It is not that every lamp explodes. The thing is, the explosions do happen from time to time and if it would be capable to cause some severe harm, even 1 out of 1000 is waaayyy too much...
Plus sometimes the explosion could be result of not primarly the lamp itself, but rather ballast failure causing severe overload (mainly relevant for series chokes, so 230V markets where frying choke increases the current; with CWA or HX such failure is nearly impossible, just because how they are designed - with most any single defect leads to current becoming lower), that is why European lamps are all rated enclosed only except very few shrouded types.

Why the danger of explosion from ballast failure was not addressed with Merc lamps ? (nearly always open highbay luminaires, they were still being installed like that well into the 80s)

Its kind of strange that even US 400w tubular lamps, which I’ve only seen on the internet, are rated for enclosed use only, same with ET18 or ED18 lamps, while the standard ones can be operated open if they are vertical.

Why the danger of explosion from ballast failure was not addressed with Merc lamps ? (nearly always open highbay luminaires, they were still being installed like that well into the 80s)

First the MVs came from the era with way looser safety culture (when the acceptable risk level was just higher than today), so many fixture designs dont treat that.
Then if you read the official safety instructions of lamp makers, they explicitely mention the risk of explosion for any HID except those explicitely open rated. They recommend protection enclosure, or to ensure the eventual explision does not cause harm (my comment on the last part: a legal loophole allowing operation of the old, open bottom street lights in rural areas where probability of someone being there during the explosion is really low). So many installations used this loophole.
Second, more technical, reason (why the above loophole is accepted) is the unsaturated vapor concept means there is not that large pressure before the arctube fails due to heat alone, so the explosion has rather limited energy and dampened by the softening arctube material (a small bubble of melred quartz on the arctube pops, the rest remains intact), so it is way less likely for the outer to not contain it.
With saturated vapor MH the pressure ruptures the quartz before it reaches temperatures any close to softening it a bit first, so you really get energetic shrapnells...

Its kind of strange that even US 400w tubular lamps, which I’ve only seen on the internet, are rated for enclosed use only, same with ET18 or ED18 lamps, while the standard ones can be operated open if they are vertical.

The tubular have the outer wall closer to the arctube, so it has less chance to contain tge explosion (the periodic restarts required are aimed to cause the arctube, if aged too much, to burst at lower pressure, so the explosion is less violent, so the larger bulb could contain it, but it is still too much for the outer being so close as with the tubular. Plus there were cases, where the outer was actually the first tube to fail and shatter, due to the high heat exposure...

The tubular have the outer wall closer to the arctube, so it has less chance to contain tge explosion (the periodic restarts required are aimed to cause the arctube, if aged too much, to burst at lower pressure, so the explosion is less violent, so the larger bulb could contain it, but it is still too much for the outer being so close as with the tubular. Plus there were cases, where the outer was actually the first tube to fail and shatter, due to the high heat exposure...

I've never seen MH lamps with arc tube explosion protector to be sold or popular here(in my country).Reason for that is that in 99% these bulbs are used in enclosed fixtures,and if bulb explodes it wouldn't fall on someones head.

The tubular lamps i seen generally feel like they have thicker and stronger walls than elliptic. But indeed the explosion will be spread over less area..

Over here many shops etc use (or were using, before LEDs) 400W MHs in open highbay luminaires. Mostly pulsestart, sometimes fitted with probestart lamps, and most of the time poorly maintained, reaching to the point of cycling and dim pink color lamps. Also, some highbay designs - especially from MS light were made so that the lamp heats directly the ballast and rises the chances of a ballast failure. I have never seen an exploded lamp in any such luminaire though - It must have been very rare occurence

The manufacturers include that warning on the lamps so that if anyone does get injured by a lamp exploding when not enclosed it means they wouldn't have grounds to make a claim for compensation.