I have a cluster of servers, each of them having 128GB or RAM and 6 x 2TB spinning disks dedicated for BlueStore OSDs.The servers also act like KVM hosts, so they are not dedicated to Ceph.In the past when using FileStore we noticed that if a server has low available memory (e.g. 10-20G), then the OSDs on this host start to do a lot more IOs than the others, generally slowing down the whole cluster. Now with BlueStore, I can see that each OSD daemon reserves around 3-4GB of memory for cache. I have reserved 5GB per OSD per server that won't be spent on VMs just to be safe.
My questions is does it matter now how much more free memory a host has for the performance and do I need to pack most VMs on hosts without OSDs like before? Or I don't need to think about that as long as I don't run into an OOM situation?
I am using Ceph 14.2.0.
