| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-02 | |||
| 12:08:15 | elodilles | ACK | |
| 12:09:49 | sean-k-mooney | i actully need to add "supprotign watcher in the sdk" to the ptg adjenda there and speak to stephen et al about the right way to do that so whtat we can also deprecate adn or remove the watcherclient python binding in a similar time frame | |
| 12:21:42 | gokhan | hi folks, I am currently testing OpenStack Epoxy with Masakari for instance high availability. In my lab environment, I am using Ceph RBD as the backend for Nova ephemeral disks and for cinder disks. I am performing "hard failure" tests by physically pulling the power plug of a compute node to trigger Masakari’s evacuation process. While Masakari successfully detects the failure and starts the instances on another node, I am facing sever | |
| 12:21:43 | gokhan | What is the recommended disk_cachemodes for Ceph RBD in production environments where power-loss-induced HA is a requirement? Is network=none the only safe path? Would switching to virtio-scsi (with hw_scsi_model and hw_disk_bus) significantly improve the "flush" command reliability during sudden failures compared to standard virtio-blk? | |
| 12:21:43 | gokhan | e filesystem corruption on almost all evacuated VMs.Upon recovery on the new node, VMs drop to emergency mode with blk_update_request: I/O error and journal corruption. I suspect network=writeback is the primary culprit during a hard power-off because volatile buffers are lost before being flushed to the Ceph OSDs. Is it a common consensus to avoid network=writeback when using Masakari or any HA driver that expects survival after a hard node crash? | |
| 12:23:59 | sean-k-mooney | gokhan: so i asume masakari is using evacuate. its a hard requiremt of using the evacuate api that the vm must be stopped on the orginal host and the admin or masikari in this case is requried to verify that before calling evacuate. by fencing i assume you mean that the phsyical host runing the vm is power off with ipmi/redfhist/pdu | |
| 12:24:28 | sean-k-mooney | network=writeback still commit all guest flushes to the backing store i.e. ceph | |
| 12:25:04 | sean-k-mooney | so if your guest os is properly flushing write then there shoudl not be any currption | |
| 12:25:43 | sean-k-mooney | or at least chanign to to none for the cache mode woudl not alter the flush behavior | |
| 12:27:21 | gokhan | thanks sean-k-mooney , Regarding your point on fencing: Yes, Masakari is indeed using the Evacuate API, and I am simulating a hard failure by physically pulling the power plug of the compute node (simulating a sudden power outage or a motherboard failure where no graceful shutdown or IPMI action is possible from the host side). | |
| 12:27:22 | sean-k-mooney | the production recomendation for ceph is hw_disk_bus=scsi hw_scsi_model=virtio-scsi and cash mode set to writeback | |
| 12:27:30 | sean-k-mooney | that is the recommendation form the ceph comunity | |
| 12:33:41 | sean-k-mooney | https://docs.ceph.com/en/latest/rbd/rbd-openstack/#image-properties https://docs.ceph.com/en/latest/rbd/rbd-openstack/#configuring-nova | |
| 12:33:52 | sean-k-mooney | by the way there si not ceph level rbd caching as well | |
| 12:34:00 | sean-k-mooney | i.e. at the ceph.conf level | |
| 12:36:22 | sean-k-mooney | i think the current recommendaions are to weriteback at the qemu level and enabling rbd cachign with ` rbd cache writethrough until flush = true` at the rbd/ceph level | |
| 12:36:54 | sean-k-mooney | with that sadi the removed the expiclty specificatgion of the livbirt cache_mode form there doc at some poitn | |
| 12:37:39 | sean-k-mooney | https://docs.ceph.com/en/mimic/rbd/rbd-openstack/#id2 | |
| 12:38:25 | sean-k-mooney | gokhan: one of hte main reasonis for usein virtio-scsi in the past was for discard/trim supprot because that was not previously implmented wehn useing virtio-blk | |
| 12:38:58 | sean-k-mooney | that gap has been resolved in the last 2-3 years in modern qemu releases | |
| 12:41:25 | gokhan | sean-k-mooney, there is no cache on ceph environment | |
| 12:41:40 | gokhan | ı will try with new configs | |
| 12:41:51 | sean-k-mooney | rbd cachign is apprenly enabled by default reading the ceph docs | |
| 12:52:03 | gokhan | sean-k-mooney, do we need to disable it or keep as is | |
| 12:53:25 | sean-k-mooney | honestly this is not something we test or can really advise on. you might get more userful feedback form the ops mailing list or operator hour session | |
| 12:53:54 | sean-k-mooney | the best i can do is point you at the offical docs from teh ceph comnuity above | |
| 12:54:43 | sean-k-mooney | well that and https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L668-L700 | |
| 12:55:10 | sean-k-mooney | a flush form the guest is flushed to ceph in all cases expct unsafe | |
| 12:55:53 | sean-k-mooney | you absolutely shoudl no use unsafe unless you hate your guests data | |
| 12:57:15 | sean-k-mooney | i woudl read that comemtn that explians the differnt modes in detail and decive which fits your performance and correctness needs | |
| 12:57:44 | sean-k-mooney | a guest shoudl never rely on writs being commited without flushing. if they do they have a bug in there software | |
| 12:58:29 | sean-k-mooney | writethough and directsync will inject extra fsync on every wite with the performance penaly that implies | |
| 13:07:23 | gokhan | disk_cachemodes=file=none,block=none,network=none | |
| 13:07:24 | gokhan | is also not worked forthis scenario. Now I am trying with recommended settings | |
| 13:13:19 | gokhan | sean-k-mooney, I find the problem. ceph is locked disk from previous host. | |
| 13:16:02 | gokhan | client.95989750 auto 125705438402288 10.x.x.x:0/1193021369 | |
| 13:16:02 | gokhan | Locker ID Address | |
| 13:16:02 | gokhan | There is 1 exclusive lock on this image. | |
| 13:16:02 | gokhan | rbd lock list cinder-volumes/volume-4263698a-ad00-472c-b4c8-62591664a8e1 | |
| 13:18:20 | gokhan | sean-k-mooney, I also saw tour response to mail list https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032256.html | |
| 14:44:52 | opendevreview | ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310 | |
| 14:44:53 | opendevreview | ribaudr proposed openstack/nova master: Tidy up pci self.flags() calls in SR-IOV functional tests https://review.opendev.org/c/openstack/nova/+/978479 | |
| 14:48:38 | opendevreview | Max proposed openstack/nova master: fix: delete attachments after rescheduling delete https://review.opendev.org/c/openstack/nova/+/974832 | |
| 15:10:09 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop wrong strict assertion of quota class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:12:25 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop wrong strict assertion of quota class class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:12:36 | tkajinam | ^^^ this is strange but it's the behavior novaclient has been testing... | |
| 15:13:07 | opendevreview | Takashi Kajinami proposed openstack/nova master: Fix wrong strict assertion of quota class class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:13:36 | opendevreview | Takashi Kajinami proposed openstack/python-novaclient master: DNM: testing https://review.opendev.org/c/openstack/python-novaclient/+/978532 | |
| 15:25:38 | Uggla | Nova meeting in ~35mn | |
| 15:52:14 | opendevreview | Takashi Kajinami proposed openstack/nova master: Fix wrong strict assertion of quota class class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:54:18 | tkajinam | hmmm so there is an issue with zuul now and jobs may be delayed for some time. | |
| 15:54:45 | tkajinam | I mean it take some time until zuul actually starts processing notifications from gerrit | |
| 15:55:02 | tkajinam | (seeing the discussion in OpenDev room in matrix | |
| 16:00:47 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:47 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:47 | opendevmeet | Meeting started Mon Mar 2 16:00:47 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:47 | Uggla | #startmeeting nova | |
| 16:00:55 | Uggla | Hello everyone | |
| 16:01:15 | elodilles | o/ | |
| 16:01:21 | bauzas | o/ | |
| 16:01:31 | sean-k-mooney | o/ is here but distracted | |
| 16:01:31 | kaisers | o/ | |
| 16:01:47 | gmaan | o/ | |
| 16:01:47 | tkajinam | o/ | |
| 16:02:08 | lajoskatona | o/ | |
| 16:03:04 | Uggla | Let's start | |
| 16:03:13 | Uggla | #topic Bugs (stuck/critical) | |
| 16:03:15 | fwiesel | o/ | |
| 16:03:23 | Uggla | #info No Critical bug | |
| 16:03:27 | tkajinam | https://bugs.launchpad.net/nova/+bug/2143057 | |
| 16:03:39 | Uggla | oh | |
| 16:03:45 | dansmith | o/ | |
| 16:03:47 | tkajinam | there is one I reported 1 hour ago (I changed its importance a few minutes ago) | |
| 16:03:56 | tkajinam | which is blocking the novaclient CI | |
| 16:04:33 | tkajinam | I've proposed a potential fix but zuul is now slow and is not starting jobs timely... | |
| 16:04:56 | Uggla | ok I have seen you message earlier about this. | |
| 16:05:00 | tkajinam | yeah | |
| 16:05:36 | Uggla | the bug sounds like something introduced with openapi. | |
| 16:05:55 | Uggla | am I right ? | |
| 16:06:06 | tkajinam | yup | |
| 16:06:15 | tkajinam | the error is caused by schema validation of responses | |
| 16:06:23 | sean-k-mooney | well that or its now showign a bug in the client | |
| 16:06:27 | tkajinam | which was recently enabled at the last phase of openapi series | |
| 16:06:41 | sean-k-mooney | ya so respocne validation is only enabeld in ci | |
| 16:06:47 | gmaan | is not the test should be fixed in novaclient? | |
| 16:06:49 | sean-k-mooney | it not inteneded for production use | |
| 16:07:04 | sean-k-mooney | and ya i am wonderign if its a novaclinet issue or not | |
| 16:07:12 | gmaan | if API doc says, default is the only supported | |
| 16:07:26 | tkajinam | but only API doc says that. that is the problem. | |
| 16:07:33 | gmaan | fake-class-2-1 -> 'default' | |
| 16:07:45 | tkajinam | I mean API doc says only default is supported but it's not actually validated in API | |
| 16:07:53 | sean-k-mooney | tkajinam: that is the actual api contract however | |
| 16:08:43 | tkajinam | I'm ok with "fixing" novaclient tests but in that case we should also fix nova to return an appropriate error to client for non-default id | |
| 16:08:53 | tkajinam | currently it returns 400 with response schema validation error | |
| 16:09:10 | sean-k-mooney | im alittel confused because i see tempst/lib/cli in that path | |
| 16:09:26 | sean-k-mooney | are these novaclient tempest tests? | |
| 16:09:38 | sean-k-mooney | "/home/zuul/src/opendev.org/openstack/python-novaclient/.tox/functional/lib/python3.12/site-packages/tempest/lib/cli/base.py" | |
| 16:10:15 | gmaan | novaclient but they use tempest base test for cli | |
| 16:10:23 | sean-k-mooney | ack | |
| 16:10:24 | gmaan | anyways, let's continue on gerrit, I wll comment there | |