What were the reasons for it..?
Get higher operating voltages, so prevent the need for high currents in the circuit, but still avoid the need for the lamp itself to see the high voltage (it means lower efficacy/shorter life/higher cost with incandescents).
Plus because the majority losses in electrical power distribution are in the wire resistance, it means all lamps get in an easy way exactly the same power, so it was possible to operate all at maximum efficacy while having their wear still under control.
It seems to me that if one lamp failed, it could take out a whole chunk of the street
No, there were bypass devices:
The lamp socket was in a type of connector, which shorts out when the lamp socket was pulled out from the fixture.
The socket side of that connector was made so the input conductors were separated by a thin paper disc.
Normally the lamp filament kept the voltage in check, so the disc remains an isolation.
But when the filament burns out, full circuit OCV becomes present across the thin disc. Becase that was usually few kV, the disc was not able to handle that voltage and it burned through, shorting the contacts and by that automatically bypassing the failed lamp.
With lamp replacement, the lamp socket, together with the burned disc was pulled from the fixture, at least to disconnect it from the high voltage (so to allow safe lamp replacement). The contacts in the fixture directly closed,keeping the circuit complete.
Then the service man replaced both the lamp, as well as the disc (and inspected the socket) and then stick the lamp socket back into the fixture.
Some makers offered lamp types with the bypass device built into the lamp assemblies (so the paper disc remained unburnt), but I don't know if that ever became wide spread (the lamps were more expensive, while the paper disc were really a cut piece of hard paper, so really dirt cheap; and the lamp internal bypass may have been not 100% efficient in preventing the paper disc from degrading/burning, so its inspection, costing the same effort so labor money, remained necessary anyway).
Since then similar bypass device (although designed for lower voltages and currents) became common in the Christmas tree lamp sets till they become replaced by LEDs. There it is in the form of an oxidized wire wrapped around the lead in wires within the lamps. The high voltage then breaks down the oxide layer, causing the wires to short through there.