| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-02 | |||
| 12:05:29 | elodilles | sean-k-mooney: can we set novaclient as retired maybe next cycle? | |
| 12:05:49 | sean-k-mooney | i was thinking more for 2027.1 | |
| 12:05:50 | elodilles | maybe that would be the most straightforward way | |
| 12:06:36 | elodilles | sean-k-mooney: i see, 2027.1 also sounds OK to me | |
| 12:06:49 | sean-k-mooney | we still have services using it so if we set that as a goal for 2027.1 that give 2 relases to complete the move but we shoudl discuss that with the wider team | |
| 12:07:01 | sean-k-mooney | watcher had it as a priorty this cycle | |
| 12:07:31 | elodilles | sean-k-mooney: this cycle the clients are already released, so i think we cannot do anything about it | |
| 12:07:42 | sean-k-mooney | ya we cant | |
| 12:07:50 | sean-k-mooney | which is fine | |
| 12:08:15 | elodilles | ACK | |
| 12:09:49 | sean-k-mooney | i actully need to add "supprotign watcher in the sdk" to the ptg adjenda there and speak to stephen et al about the right way to do that so whtat we can also deprecate adn or remove the watcherclient python binding in a similar time frame | |
| 12:21:42 | gokhan | hi folks, I am currently testing OpenStack Epoxy with Masakari for instance high availability. In my lab environment, I am using Ceph RBD as the backend for Nova ephemeral disks and for cinder disks. I am performing "hard failure" tests by physically pulling the power plug of a compute node to trigger Masakari’s evacuation process. While Masakari successfully detects the failure and starts the instances on another node, I am facing sever | |
| 12:21:43 | gokhan | e filesystem corruption on almost all evacuated VMs.Upon recovery on the new node, VMs drop to emergency mode with blk_update_request: I/O error and journal corruption. I suspect network=writeback is the primary culprit during a hard power-off because volatile buffers are lost before being flushed to the Ceph OSDs. Is it a common consensus to avoid network=writeback when using Masakari or any HA driver that expects survival after a hard node crash? | |
| 12:21:43 | gokhan | What is the recommended disk_cachemodes for Ceph RBD in production environments where power-loss-induced HA is a requirement? Is network=none the only safe path? Would switching to virtio-scsi (with hw_scsi_model and hw_disk_bus) significantly improve the "flush" command reliability during sudden failures compared to standard virtio-blk? | |
| 12:23:59 | sean-k-mooney | gokhan: so i asume masakari is using evacuate. its a hard requiremt of using the evacuate api that the vm must be stopped on the orginal host and the admin or masikari in this case is requried to verify that before calling evacuate. by fencing i assume you mean that the phsyical host runing the vm is power off with ipmi/redfhist/pdu | |
| 12:24:28 | sean-k-mooney | network=writeback still commit all guest flushes to the backing store i.e. ceph | |
| 12:25:04 | sean-k-mooney | so if your guest os is properly flushing write then there shoudl not be any currption | |
| 12:25:43 | sean-k-mooney | or at least chanign to to none for the cache mode woudl not alter the flush behavior | |
| 12:27:21 | gokhan | thanks sean-k-mooney , Regarding your point on fencing: Yes, Masakari is indeed using the Evacuate API, and I am simulating a hard failure by physically pulling the power plug of the compute node (simulating a sudden power outage or a motherboard failure where no graceful shutdown or IPMI action is possible from the host side). | |
| 12:27:22 | sean-k-mooney | the production recomendation for ceph is hw_disk_bus=scsi hw_scsi_model=virtio-scsi and cash mode set to writeback | |
| 12:27:30 | sean-k-mooney | that is the recommendation form the ceph comunity | |
| 12:33:41 | sean-k-mooney | https://docs.ceph.com/en/latest/rbd/rbd-openstack/#image-properties https://docs.ceph.com/en/latest/rbd/rbd-openstack/#configuring-nova | |
| 12:33:52 | sean-k-mooney | by the way there si not ceph level rbd caching as well | |
| 12:34:00 | sean-k-mooney | i.e. at the ceph.conf level | |
| 12:36:22 | sean-k-mooney | i think the current recommendaions are to weriteback at the qemu level and enabling rbd cachign with ` rbd cache writethrough until flush = true` at the rbd/ceph level | |
| 12:36:54 | sean-k-mooney | with that sadi the removed the expiclty specificatgion of the livbirt cache_mode form there doc at some poitn | |
| 12:37:39 | sean-k-mooney | https://docs.ceph.com/en/mimic/rbd/rbd-openstack/#id2 | |
| 12:38:25 | sean-k-mooney | gokhan: one of hte main reasonis for usein virtio-scsi in the past was for discard/trim supprot because that was not previously implmented wehn useing virtio-blk | |
| 12:38:58 | sean-k-mooney | that gap has been resolved in the last 2-3 years in modern qemu releases | |
| 12:41:25 | gokhan | sean-k-mooney, there is no cache on ceph environment | |
| 12:41:40 | gokhan | ı will try with new configs | |
| 12:41:51 | sean-k-mooney | rbd cachign is apprenly enabled by default reading the ceph docs | |
| 12:52:03 | gokhan | sean-k-mooney, do we need to disable it or keep as is | |
| 12:53:25 | sean-k-mooney | honestly this is not something we test or can really advise on. you might get more userful feedback form the ops mailing list or operator hour session | |
| 12:53:54 | sean-k-mooney | the best i can do is point you at the offical docs from teh ceph comnuity above | |
| 12:54:43 | sean-k-mooney | well that and https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L668-L700 | |
| 12:55:10 | sean-k-mooney | a flush form the guest is flushed to ceph in all cases expct unsafe | |
| 12:55:53 | sean-k-mooney | you absolutely shoudl no use unsafe unless you hate your guests data | |
| 12:57:15 | sean-k-mooney | i woudl read that comemtn that explians the differnt modes in detail and decive which fits your performance and correctness needs | |
| 12:57:44 | sean-k-mooney | a guest shoudl never rely on writs being commited without flushing. if they do they have a bug in there software | |
| 12:58:29 | sean-k-mooney | writethough and directsync will inject extra fsync on every wite with the performance penaly that implies | |
| 13:07:23 | gokhan | disk_cachemodes=file=none,block=none,network=none | |
| 13:07:24 | gokhan | is also not worked forthis scenario. Now I am trying with recommended settings | |
| 13:13:19 | gokhan | sean-k-mooney, I find the problem. ceph is locked disk from previous host. | |
| 13:16:02 | gokhan | rbd lock list cinder-volumes/volume-4263698a-ad00-472c-b4c8-62591664a8e1 | |
| 13:16:02 | gokhan | There is 1 exclusive lock on this image. | |
| 13:16:02 | gokhan | Locker ID Address | |
| 13:16:02 | gokhan | client.95989750 auto 125705438402288 10.x.x.x:0/1193021369 | |
| 13:18:20 | gokhan | sean-k-mooney, I also saw tour response to mail list https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032256.html | |
| 14:44:52 | opendevreview | ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310 | |
| 14:44:53 | opendevreview | ribaudr proposed openstack/nova master: Tidy up pci self.flags() calls in SR-IOV functional tests https://review.opendev.org/c/openstack/nova/+/978479 | |
| 14:48:38 | opendevreview | Max proposed openstack/nova master: fix: delete attachments after rescheduling delete https://review.opendev.org/c/openstack/nova/+/974832 | |
| 15:10:09 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop wrong strict assertion of quota class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:12:25 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop wrong strict assertion of quota class class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:12:36 | tkajinam | ^^^ this is strange but it's the behavior novaclient has been testing... | |
| 15:13:07 | opendevreview | Takashi Kajinami proposed openstack/nova master: Fix wrong strict assertion of quota class class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:13:36 | opendevreview | Takashi Kajinami proposed openstack/python-novaclient master: DNM: testing https://review.opendev.org/c/openstack/python-novaclient/+/978532 | |
| 15:25:38 | Uggla | Nova meeting in ~35mn | |
| 15:52:14 | opendevreview | Takashi Kajinami proposed openstack/nova master: Fix wrong strict assertion of quota class class set id https://review.opendev.org/c/openstack/nova/+/978494 | |
| 15:54:18 | tkajinam | hmmm so there is an issue with zuul now and jobs may be delayed for some time. | |
| 15:54:45 | tkajinam | I mean it take some time until zuul actually starts processing notifications from gerrit | |
| 15:55:02 | tkajinam | (seeing the discussion in OpenDev room in matrix | |
| 16:00:47 | Uggla | #startmeeting nova | |
| 16:00:47 | opendevmeet | Meeting started Mon Mar 2 16:00:47 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:47 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:47 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:55 | Uggla | Hello everyone | |
| 16:01:15 | elodilles | o/ | |
| 16:01:21 | bauzas | o/ | |
| 16:01:31 | kaisers | o/ | |
| 16:01:31 | sean-k-mooney | o/ is here but distracted | |
| 16:01:47 | tkajinam | o/ | |
| 16:01:47 | gmaan | o/ | |
| 16:02:08 | lajoskatona | o/ | |
| 16:03:04 | Uggla | Let's start | |
| 16:03:13 | Uggla | #topic Bugs (stuck/critical) | |
| 16:03:15 | fwiesel | o/ | |
| 16:03:23 | Uggla | #info No Critical bug | |
| 16:03:27 | tkajinam | https://bugs.launchpad.net/nova/+bug/2143057 | |
| 16:03:39 | Uggla | oh | |
| 16:03:45 | dansmith | o/ | |
| 16:03:47 | tkajinam | there is one I reported 1 hour ago (I changed its importance a few minutes ago) | |
| 16:03:56 | tkajinam | which is blocking the novaclient CI | |
| 16:04:33 | tkajinam | I've proposed a potential fix but zuul is now slow and is not starting jobs timely... | |
| 16:04:56 | Uggla | ok I have seen you message earlier about this. | |
| 16:05:00 | tkajinam | yeah | |
| 16:05:36 | Uggla | the bug sounds like something introduced with openapi. | |
| 16:05:55 | Uggla | am I right ? | |
| 16:06:06 | tkajinam | yup | |
| 16:06:15 | tkajinam | the error is caused by schema validation of responses | |
| 16:06:23 | sean-k-mooney | well that or its now showign a bug in the client | |
| 16:06:27 | tkajinam | which was recently enabled at the last phase of openapi series | |
| 16:06:41 | sean-k-mooney | ya so respocne validation is only enabeld in ci | |
| 16:06:47 | gmaan | is not the test should be fixed in novaclient? | |
| 16:06:49 | sean-k-mooney | it not inteneded for production use | |
| 16:07:04 | sean-k-mooney | and ya i am wonderign if its a novaclinet issue or not | |
| 16:07:12 | gmaan | if API doc says, default is the only supported | |
| 16:07:26 | tkajinam | but only API doc says that. that is the problem. | |
| 16:07:33 | gmaan | fake-class-2-1 -> 'default' | |
| 16:07:45 | tkajinam | I mean API doc says only default is supported but it's not actually validated in API | |