Also you would think if the car won't unlock when handle pulled that the button on the remote wouldn't work either, yet it always does.
Unlocking by the handle needs the communication to work both in the LW (from the car to the key, to send authentication query), as well in the UHF (from the key to the car, to send the response).
If you press the button, all goes from the key to the car, so just using the UHF link.
So when there is something making the LW link not working properly, you get exactly the behavioryou describe. And because how the LW link is designed (using signal strength as a means to check the distance), the "something" could be anything from a wiring fault influencing the transmitter antenna loop, broken receiver antenna, till just weak battery in the receiver (the key).
The UHF link (key to car) has way less predictable signal paths, so it is designed with plenty of margin, so until the microcontroller in the key is still assembling correct pattern, the transmission works even with very weak battery.
I did find there is a spot in the center console I can put the remote and even if the battery is dead the car would still start.
There is direct RFID type channel, which works only over few cm, but does not need any battery in the key.
If that is working correctly, the odds are (if the maker was cheap and so uses common antenna for the RFID, as well as the LW link inside of the key), the key ferrite antenna is fine so the culprit could really be just the battery being weak.
Before replacing the battery, make sure you will be able to resynchronize the key with the car. One of the security method used there is based on a pseudorandom generator scrambling the data send over the UHF, so any command message wont be valid again (usually with 2^32 or even longer cycle). In order to work, these generators must be in sync between the car and the key. Removing the battery may reset it in the key, but the car side remains in its progressed state. So the remote wont work, until you align them back together. The procedure is exactly described in the service manual, sometimes even in the users manual, but it tend to differ among car brands...