| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-23 | |||
| 15:21:34 | kashyap | He's a fixed a similar-sounding issue as above --^ It was also triggerred by OpenStack CI | |
| 15:21:42 | kashyap | s/a fixed/fixed/ | |
| 15:22:07 | sean-k-mooney | gibi: the main delta there being virtio-ballon vs virtio-blk | |
| 15:38:42 | kashyap | sean-k-mooney: Also see this one: https://bugs.launchpad.net/nova/+bug/1737625 | |
| 15:38:42 | openstack | Launchpad bug 1737625 in OpenStack Compute (nova) "live migration failed with "Failed to load virtio-blk:virtio" (libvirt/qemu)" [Undecided,Confirmed] | |
| 15:38:58 | kashyap | Especially my comment#2, FWIW | |
| 15:45:15 | openstackgerrit | Merged openstack/nova master: Fix the instance_uuid for test https://review.opendev.org/c/openstack/nova/+/763171 | |
| 15:45:20 | sean-k-mooney | gibi: by the way this might be of interest to you https://bugzilla.redhat.com/show_bug.cgi?id=1895702 | |
| 15:45:20 | openstack | bugzilla.redhat.com bug 1895702 in openstack-neutron "Nova – VM is created in spite of the fact that used port is missing fixed IP" [High,New] - Assigned to rhos-maint | |
| 15:46:05 | openstackgerrit | Merged openstack/nova master: doc: Update the PTL guide https://review.opendev.org/c/openstack/nova/+/762727 | |
| 15:46:38 | sean-k-mooney | kashyap we might want to combine these or at least cross link them | |
| 15:47:01 | gibi | sean-k-mooney: interesing edge case where nova allows booting without IP | |
| 15:47:24 | kashyap | sean-k-mooney: So ... the ideal person to help debug this would be Stefan (he's on PTO), who is the 'virtio-blk' expert | |
| 15:47:26 | sean-k-mooney | gibi: ya i think its actully a neutron bug but you were asking about could we enable that behavior | |
| 15:47:46 | kashyap | That said, bugs like these, deep in the bowels of QEMU, require a debug QEMU with probes installed in the environment ... | |
| 15:47:52 | gibi | sean-k-mooney: yes, I got asked about it internally | |
| 15:48:03 | kashyap | ... and pray to the JuJu at the bottom of the sea that the bug reproduces | |
| 15:48:06 | sean-k-mooney | kashyap: if it starts blocking the ci then we will need to take more imediate action but until then i think it can wait till they are back form pto | |
| 15:48:37 | kashyap | sean-k-mooney: I highly doubt it can "block"; I just read Dave's explanation on the rare occurrence it can hit | |
| 15:48:45 | kashyap | As the last known occurrence in the CI seems to be 3 years ago | |
| 15:49:02 | kashyap | (But I'll comment further in the bug) | |
| 15:49:30 | gibi | yeah, i have no problem ignoring this qemu issue if it happens once in two years | |
| 15:50:01 | sean-k-mooney | yep same | |
| 15:50:15 | sean-k-mooney | we just know that it can fail this way | |
| 15:50:18 | kashyap | sean-k-mooney: Also, what's happening in the guest? | |
| 15:50:32 | kashyap | Is it just a garden-variety CirrOS instance just idling? | |
| 15:50:35 | sean-k-mooney | its just a migtrion of a cirror guest | |
| 15:50:41 | sean-k-mooney | yep | |
| 15:51:20 | sean-k-mooney | so beyound the ocational dhcp request or other idel work it should be doing nothing most of the time | |
| 15:54:55 | kashyap | sean-k-mooney: Dave says if it's a migrate-during-a-reboot, it could be a potential problem | |
| 15:54:58 | kashyap | That said ... | |
| 15:55:38 | kashyap | sean-k-mooney: gibi: I learn from Dave that we get that "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. | |
| 15:55:44 | kashyap | And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. | |
| 15:55:52 | kashyap | (I'll add a comment in the bug from 2018 with notes) | |
| 15:58:22 | gibi | kashyap: thanks | |
| 16:10:36 | stephenfin | bauzas: Think you could take a look at this this week? https://review.opendev.org/c/openstack/nova/+/756549 | |
| 16:10:49 | bauzas | stephenfin: sure, will look | |
| 16:10:53 | stephenfin | ta | |
| 16:14:13 | openstackgerrit | Merged openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication https://review.opendev.org/c/openstack/nova-specs/+/759828 | |
| 16:28:53 | openstackgerrit | Merged openstack/nova master: Ignore InstanceNotFound exception when host rejoin cluster https://review.opendev.org/c/openstack/nova/+/757053 | |
| 16:39:14 | stephenfin | dansmith: I know it's your final day before the new year, but if you've a chance to provide guidance at [1] I'd appreciate it. Would like to close that out this year :) https://review.opendev.org/c/openstack/nova/+/759114/3/nova/db/sqlalchemy/models.py#1212 | |
| 16:40:19 | dansmith | stephenfin: ack | |
| 16:50:11 | kashyap | gibi: sean-k-mooney: I've outlined some notes here after my chat with QEMU migration maintainer, FWIW: https://bugs.launchpad.net/nova/+bug/1737625/comments/4 | |
| 16:50:11 | openstack | Launchpad bug 1737625 in OpenStack Compute (nova) "live migration failed with "Failed to load virtio-blk:virtio" (libvirt/qemu)" [Undecided,Confirmed] | |
| 16:51:03 | sean-k-mooney | kashyap: we might want to use the other bug | |
| 16:51:13 | sean-k-mooney | and close that as a duplicate | |
| 16:51:35 | kashyap | sean-k-mooney: Isn't this more accurate? It is talking about 'virtio-blk' and the Gate issue also is virtio-blk | |
| 16:51:44 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1761798 | |
| 16:51:44 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 16:51:56 | sean-k-mooney | kashyap: no i dont think it is i think its the same underlying issue | |
| 16:52:56 | kashyap | In any case, I'm copying my comment into this one, too: https://bugs.launchpad.net/nova/+bug/1761798 | |
| 16:52:56 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 18:14:26 | openstackgerrit | Merged openstack/nova master: virt: Remove 'change_instance_metadata' API https://review.opendev.org/c/openstack/nova/+/749316 | |
| 18:42:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Remove 'reset_network' API https://review.opendev.org/c/openstack/nova/+/749315 | |
| 18:42:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Remove 'get_all_bw_counters' API https://review.opendev.org/c/openstack/nova/+/749312 | |
| 18:42:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'BandwidthUsage', 'BandwidthUsageList' https://review.opendev.org/c/openstack/nova/+/759114 | |
| 18:42:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Drop models for removed services, features https://review.opendev.org/c/openstack/nova/+/698706 | |
| 19:12:20 | sean-k-mooney | stephenfin: ah those are related to xenapi removal | |
| 19:16:50 | sean-k-mooney | i tought it was related to the compute_monitors config intially | |
| 19:16:57 | sean-k-mooney | https://github.com/openstack/nova/blob/120d866df7ab81b666c0e43b97705a90483533bc/nova/conf/compute.py#L107-L126 | |
| 19:24:02 | sean-k-mooney | stephenfin: by the way we talked about the metrics weigher and weighing on things that were not compute weights in the past | |
| 19:24:06 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/monitors/__init__.py#L40-L48 | |
| 19:24:17 | sean-k-mooney | the monitors are a stevaldor entry point | |
| 19:24:30 | sean-k-mooney | and the numa memory bandwtih ones were actully plugins not in tree | |
| 19:24:57 | sean-k-mooney | https://github.com/openstack/nova/blob/c250aca00ba89b6fad86bfaebb3083e9f5c55f3b/nova/compute/monitors/__init__.py#L44 the doc used to reference them years ago | |
| 19:28:43 | sean-k-mooney | well it was propsoed upstream https://review.opendev.org/c/openstack/nova/+/270344 | |
| 19:28:53 | sean-k-mooney | but it was not merged | |
| 19:31:20 | sean-k-mooney | https://review.opendev.org/c/openstack/nova-specs/+/180983 was the spec | |
| 21:21:27 | sean-k-mooney | gibi: stephenfin i just realised that https://review.opendev.org/c/openstack/nova/+/653506 is still not merged but its been so long that i have lost most of the context. its still proably should be fixed but im not sure when ill get to it. hopefully soon but i tought this was merged years ago... | |
| #openstack-nova - 2020-11-24 | |||
| 00:10:54 | gmann | gibi: stephenfin: can you check this sample config fix- https://review.opendev.org/c/openstack/nova/+/762721 | |
| 00:49:36 | brinzhang | gibi, dansmith: do you have time to review the cyborg shelve/unshelve support patch, it has addressed all your patch https://review.opendev.org/c/openstack/nova/+/729563 | |
| 01:38:09 | openstackgerrit | DatLQ proposed openstack/nova stable/rocky: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/c/openstack/nova/+/763898 | |
| 01:58:13 | openstackgerrit | DatLQ proposed openstack/nova stable/rocky: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/c/openstack/nova/+/763898 | |
| 05:51:22 | openstackgerrit | Kevin Zhao proposed openstack/nova master: Support Cpu Compararion on Aarch64 Platform https://review.opendev.org/c/openstack/nova/+/763928 | |
| 07:20:24 | openstackgerrit | Jorhson Deng proposed openstack/nova-specs master: Add instance_state to ensure volume attachment successful https://review.opendev.org/c/openstack/nova-specs/+/758920 | |
| 08:05:31 | openstackgerrit | MIKI Nobuhiro proposed openstack/nova-specs master: Add IP address to libvirt guest metadata https://review.opendev.org/c/openstack/nova-specs/+/760750 | |
| 08:26:25 | openstackgerrit | Nobuhiro MIKI proposed openstack/nova-specs master: Add IP address to libvirt guest metadata https://review.opendev.org/c/openstack/nova-specs/+/760750 | |
| 08:35:59 | openstackgerrit | Jorhson Deng proposed openstack/nova-specs master: Add instance_state to ensure volume attachment successful https://review.opendev.org/c/openstack/nova-specs/+/758920 | |
| 10:46:29 | openstackgerrit | Xinran WANG proposed openstack/nova-specs master: SRIOV Enabled Nic Support Specification https://review.opendev.org/c/openstack/nova-specs/+/742785 | |
| 11:09:34 | lyarwood | gibi: morning, could I pester you for a review of https://review.opendev.org/c/openstack/nova/+/756434 - RH owner and stephenfin has already ACK'd so I didn't want to +W it without further review. | |
| 11:09:55 | gibi | lyarwood: ack, looking | |
| 11:10:06 | lyarwood | gibi: thanks | |
| 11:11:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'BandwidthUsage', 'BandwidthUsageList' https://review.opendev.org/c/openstack/nova/+/759114 | |
| 11:11:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Drop models for removed services, features https://review.opendev.org/c/openstack/nova/+/698706 | |
| 11:14:21 | gibi | lyarwood: +a | |
| 11:16:11 | lyarwood | gibi: cheers | |
| 11:54:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make PCI claim NUMA aware during live migration https://review.opendev.org/c/openstack/nova/+/748453 | |
| 12:22:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/761923 | |
| 12:22:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/761924 | |
| 12:28:45 | gibi | elod: ^^ did the squash | |
| 12:43:33 | sean-k-mooney | gibi: stephenfin since the autor of https://review.opendev.org/c/openstack/nova-specs/+/760750 ignored my attempt to colabarte and have just one spec i want to get your input on "do i really need a spec" | |
| 12:44:11 | sean-k-mooney | basically i was plannign to update the libvirt metadata with teh flavor extra specs and image proeprties via a specless blueprint | |
| 12:44:44 | sean-k-mooney | this is an internal api used only for debugging so i kind of would prefer to just propose a patch and debate the implemeantion there | |
| 12:45:19 | sean-k-mooney | i can write a spec but its not an api change so it feels kind fo heavy weight to me | |
| 12:46:42 | stephenfin | I think gibi's the only person that can really answer that | |
| 12:47:05 | gibi | do you remember why we asked for a spec for the first place? Now it would be strange to treate the two features differently | |
| 12:47:33 | stephenfin | I think because there were a few design questions and implications for things like attach/detach | |
| 12:48:26 | sean-k-mooney | i can write a spec but the only time the flavor and image will change are resize and rebuild where im regenreating the xml anyway | |
| 12:48:44 | gibi | and resize while changes the flavor it does regen... ^^ | |
| 12:49:09 | gibi | then I'm OK with a specless bp | |