Keras consumes too much ram instead of gpu

I have a data of 10 Gb in ram and the shape of data is (10000,3000,61). I am training a lstm model. But when I start training with batch size of 10242 the ram usage reach upto 50 GB, but GPU memory is half consumed. If I further increase the batch size , the system get hanged because of 100% ram consumption. Simply I am wasting 50% of my gpu capacity, because with a batch size of 10242, only 5.8 gb gpu is used and I have 12 gb gpu.