At the LG, there are lots of statements that one light source is more/less efficient than the other.
So I'd like to discuss the correct way of comparing those efficiencies.

1. Integrated vs. external ballast
I think this is the main point influencing lm/W values. I suppose that CFL, retrofit LED bulbs etc. (integrated ballast) have their power consumption (e.g. 23W) specified including the ballast while standard fluorescents, HID etc. without the ballast.
Thus, light sources with integrated ballasts are at a disadvantage.

2. Type of the ballast
Some light sources can work both with magnetic and electronic ballasts. Magnetic ballasts can heat a lot so they decrease actual lm/W values more than electronic ones.
LED drivers seem to have lower losses than fluorescent/HID ballasts (not sure about that).

3. Warm-up time
HIDs are usually burning for several hours periods so the few minutes warm-ups with increased power consumption aren't relevant.
CFLs, on the other hand, are not seldom used for tens-of-seconds periods (they don't even reach the full luminous power) so their actual lm/W efficiencies are way lower than specified.

4. Stand-by power consumption
Some LED based table lamps have their switches at their low voltage cable meaning the transformer (switched source) and the driver are being powered permanently. If this is for example a 3W lamp burning only half an hour daily (23.5 hours in standby) even the minimum power consumption in standby can make this light source very ineffective.

5. Making omnidirectional light source directional (and vice versa)
Even with metallic reflector, making the HID or fluorescent illuminate just the street below means extra losses which may decrease the actual lm/W significantly. If the lamppost isn't too high, LEDs don't require any optics at all.

6.Reactive energy (cos-phi)
Magnetic ballasts are known for low cos phi values that should be compensated. At home, this is not a problem for the end user as electricity meters register only resistive energy. But the reactive energy must be compensated somewhere. Compensating and distributing reactive energy add extra losses too.

7. Colour temperature and CRI
Most of light sources is for humans. (Other applications are horticulture, reptile lamps, stage lighting for cameras etc.) Human eye is more sensitive to some colours than to others. Thus lm/W efficiency, even calculated precisely, isn't the only factor of the light source efficiency.

Conclusion: There are various factors that must be considered before one can say which light source is more efficient. Your comments?

You've made some very good (and, to my knowledge, correct) points. The only thing I'll add is that electronic ballasts can also have a low power factor. Every electronic CFL I've measured so far had a power factor of around 0.55.

There are electronic ballasts with high power factor (Programmed start ballasts for fluorescent lamps, with an active power factor correction, that have a PFC of 1.00)

The power factor of virtually any electronic used to be quite low, the 0,55 isn't that low number for a simple rectifier-filter. Most computers from the 90's and basically all semiconductor based TV's (I mean semiconductors except the CRT; the tube based had higher power factor just because the tubes consumed a lot of power for heaters) had power factor of just 0.2..0.3 on the mains input. So most of the ballasts are way better than that.
It is only past the year 2000, when the power factor became a concern, so a correction stage become mandated, initially for just the highest power devices, but later on the power border had moved down (today it is around 40W).
With the not corrected rectifier the power factor is directly tied to the ripple on the rectified voltage: Higher the filtering capacitor, lower the ripple and power factor and better the ability to cover mains cuts and vice versa.
As the lighting is required to keep the power factor under control, it means the DC voltage is poorly filtered (with the PF=0.55 ballasts the voltage vary between 240..325V on a 230V mains, what mean brightness ripple around 50%; depends on the ability of the inverter to compensate the voltage variation influence onto the actual lamp power), so yield lamp 100/120Hz flicker.
As with higher arc voltage fluorescents the inverter is already marginal on voltages, the flicker become worse than on a low wattage types with low arc voltage (where the inverter can easily compensate the voltage variation) and/or requires te introduction of the power factor correction stage (to both boost the voltage, so there would be more headroom for the inverter, as well as allow higher value of the filtering capacitor without causing lower power factor).

@TheMaritimeMan: I'm no expert at all but I like to have things in order.

@Medved: It sounds very interesting but I'm afraid I don't follow you...

"As the lighting is required to keep the power factor under control, it means the DC voltage is poorly filtered..."
When do we need a rectifier? I suppose not with fluorescents where AC voltage is no problem. Multi LED lamps can operate with both halves of the sine period without a rectifier. You mean a single LED lamp...?

Do I understand well that those lamps with filtered rectifier in fact compensate the power factor for other magnetically ballasted lamps, motors etc.? That's not bad, is it?

"As the lighting is required to keep the power factor under control, it means the DC voltage is poorly filtered..."
When do we need a rectifier? I suppose not with fluorescents where AC voltage is no problem. Multi LED lamps can operate with both halves of the sine period without a rectifier. You mean a single LED lamp...?

Beside few exceptions (phase cut dimmers), virtually electronic works on DC, include the electronic ballasts, either for fluorescents, LED's, HID's, even the electronic "230/12V trancsformer" for the 12V halogens,...
So any electronic ballast has to first convert the mains AC to the DC needed to supply the main ballasting circuit.

The rectified DC voltage is then chopped by the inverter stage to create the high frequency for the final lamp circuit (with LED and HID that mean second rectification). Now as the DC voltage is not 100% filtered, it means it vary over the mains cycle. Because the mains cycle is many times longer than the high frequency, the generated high frequency become amplitude modulated by that ripple. But that mean the HF output to the lamp is varying. And it is this variation, which is then causes the lamp to flicker, more variations means more flicker and vice versa. And when there would be no filter on the rectified DC, the voltage will completely disappear, so will the inverter output. But with most of the circuits, the variation could be suppressed to some extend by modifying other inverter output parameters like frequency and/or duty ratio. But the working range of these methods is rather limited, so for continuous operation you need some storage (filter), what will cover the parts of the mains cycle, when the mains is insufficient.
With fluorescents the aim is to keep the discharge burning without interruption, it needs just reduced power to keep the arc alive as the minimum, so the filter capacitor could be minimalistic.
For LED's the interruption is no problem, so you do not need the filter at all, so you may design the inverter with (followed by a second rectifier, so forms a DCDC converter) so, it's input current follows the voltage, so in front of the rectifier it looks like very high power factor.
With the incandescent "transformer" the lamp don't care and it is by itself resistive, so it is fed directly by the modulated HFAC. It has the benefit of inherent high power factor, because the lamp current follows the lamp voltage, so does the inverter input current vs the rectified voltage.

But as the power to the lamp vary, the lamp flickers. Sometimes it is just annoying (fluorescents, LED), but sometimes it affect the lamp performance (HID), so the DC has to be filtered quite well and the inverter has to fully compensate that by modulating it's duty or frequency.

Do I understand well that those lamps with filtered rectifier in fact compensate the power factor for other magnetically ballasted lamps, motors etc.?

No. The non-unity power factor of magnetic ballasts is of fundamentally other nature than the non-unity power factor of the rectifiers. The thing is, to have really a unity power factor, the voltage and current gave to match in both shape and phase (as it happens on a plain resistor). As the mains is a sinewave, for unity power factor you need the load to draw sinewave input current without any phase shift from the voltage.
The concept of "phase shifts" and/or "complex number" is not even how the physics reality works, but it is just a mathematical aid (a substitution) to simplify the equations for a "special case" valid only for linear circuits with steady harmonic voltages/currents, all at the same frequency.
Well, the 99% of the AC power distribution does or could be simplified so to meet the criteria of the "linear circuit with steady state harmonic signals at a single frequency", and this mathematical simplification is so handy (it releaves you from complex derivation and integration math), so widely used and it could be used even without the knowledge of real math behind, you have to be always aware it is valid just for a special case, so before using it, you have to make sure all the signals meet the conditions or at least you know, how to mathematically convert them to their equivalents and what are the limitations of that conversion.

So the magnetic ballasts with a lamp could be simplified to a "linear system with steady state harmonic signals", so you can use that aid.
But for the nonlinear rectifier the current is not harmonic, so it means terms like "phase shift" or "reactive power" have no meaning at all, you have to stick with just the tedious operations of "derivation" and "integration".

Thanks, Medved, for the long explanation! As for the electronic ballast principle, I should have checked Wikipedia first.

@"But as the power to the lamp vary, the lamp flickers. Sometimes it is just annoying (fluorescents, LED)..." If you still mean that 100/120Hz flicker here, I haven't seen an electronically ballasted fluorescent to flicker visibly while there are LED lamps where the flicker is almost unbearable (edges of moving objects are split distinctly).

Note: When I looked at the Wikipedia page above, there is an "ANSI Ballast factor" mentioned there. I think it should also be considered when talking about lamp efficiencies.

@"But as the power to the lamp vary, the lamp flickers. Sometimes it is just annoying (fluorescents, LED)..." If you still mean that 100/120Hz flicker here, I haven't seen an electronically ballasted fluorescent to flicker visibly while there are LED lamps where the flicker is almost unbearable (edges of moving objects are split distinctly).

The flicker come from both, but with fluorescents it is of smaller amplitude, so it is way less visible, because fluorescents have to use at least some filtering to prevent the arc from extinguishing when the mains voltage itself is not sufficient. The reason is the tube lifetime: The reignition stress the cathodes, so they wear out quicker, so there will be no benefit from the HF drive.
LED's do not have this wear problem, so the filter could be omitted.
Now it looks like it is omitted just for the cost reduction, but there is a very good technical reason behind:
The filter need a capacitor of a quite high capacitance, so it is able to supply the ballast for some time (~5ms or so). For such high capacitances packed into that small space we have only one (principal) technology: Electrolytic capacitors. But these components are quite problematic from the lifetime and reliability point of view when they have to operate at high temperatures and/or high ripple current levels. And with ballasts they are exposed to both at the same time. The problem is, they need the water to be present all the time. But with high temperatures the water would boil at normal temperatures, so the vessel of these high temperature power components has to be pressure sealed. And the sealing is there the weakest point: It tend to leak after some time, so allow the water to evaporate and that mean capacitor failure.
Another reason is, the high capacitance behind the rectifier means low power factor. So when higher PF is needed, you need a power factor corrector, quite complicated stuff. And again another complexity to host a fault. Without the filter the power factor will be inherently high.
So the easiest way to avoid these problems is to not use the electrolytic capacitors, unless they are really necessary.
With hot cathode fluorescents they are necessary, because otherwise the life of the lamp will suffer, so there is no real way around.
But for CCFL the periodic reignition is of no problem, so with CCFL ballasts they are usually not used.
With older LED's they had to be used, because the ballast controllers (based on DC output SMPS architectures) had long startup time, so the DC supply had to be kept alive. Quite silly limitation...
The modern ballast controller IC's are designed to cope with the DC input interruptions, moreover are able to control the transferred power so, the power factor is nearly unity. So with these the electrolytics could be omitted, so all the related problems are gone. The only consequence is, the LED does flicker at the double of the mains frequency.
Now as that did all discharge on magnetic ballasts all the time they were used on AC mains and (except few rather special applications) it was not considered as big problem, so it is assumed it won't be problem with the LED's either.
Indeed, it appears like a step back from the comfort perspective, but the need for longer life and compact designs (so operating at higher temperatures) is a stronger argument asking to not use such problematic components.