| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-16 | |||
| 16:21:12 | bauzas | but let's continue discussing this in the proposal, I don't wanna drag the whole attention by now | |
| 16:21:19 | sean-k-mooney | bauzas: for one off patches i think the expectaion shoudl still be on use to watch the patchs come in and help them | |
| 16:21:28 | sean-k-mooney | rather ten assuemign they will use any tools we provide | |
| 16:21:49 | bauzas | sean-k-mooney: yeah but then how to discover them ? | |
| 16:22:08 | bauzas | either way, let's discuss this by Gerrit :p | |
| 16:22:10 | sean-k-mooney | well if its a similar time zone i watch for teh irc bot commeting for the patches | |
| 16:22:24 | sean-k-mooney | if i dont recognise it or the name i open it | |
| 16:22:49 | sean-k-mooney | and then one of use can request the reqview priority in gerrirt or publicise the patch to others | |
| 16:23:15 | bauzas | that's one direction | |
| 16:23:46 | sean-k-mooney | if there is something in gerrit i can set im happy to do that on patches when i think they are ready otherwise ill just ping them to ye as i do now | |
| 16:23:55 | bauzas | either way, we have a large number of items for the open discussion topic, so let's move on | |
| 16:24:01 | sean-k-mooney | ack | |
| 16:24:09 | bauzas | #topic Stable Branches | |
| 16:24:23 | bauzas | elodilles: fancy copy/pasting or do you want me to do so ? | |
| 16:24:35 | elodilles | either way is OK :) | |
| 16:25:42 | bauzas | I can do it | |
| 16:25:44 | bauzas | #info stable gates' status look OK, no blocked branch | |
| 16:25:50 | bauzas | #info final ussuri nova package release was published (21.2.4) | |
| 16:25:55 | bauzas | #info ussuri-em tagging patch is waiting for final python-novaclient release patch to merge | |
| 16:26:00 | bauzas | #link https://review.opendev.org/c/openstack/releases/+/817930 | |
| 16:26:05 | bauzas | #link https://review.opendev.org/c/openstack/releases/+/817606 | |
| 16:26:09 | bauzas | #info intermittent volume detach issue: afaik Lee has an idea and started to work on how it can be fixed: | |
| 16:26:14 | bauzas | #link https://review.opendev.org/c/openstack/tempest/+/817772/ | |
| 16:26:19 | bauzas | any question ? | |
| 16:26:25 | elodilles | thanks :) | |
| 16:27:40 | bauzas | looks like none | |
| 16:27:47 | gibi | the volume detach issue feel more an more like not related to detach | |
| 16:27:47 | bauzas | #topic Sub/related team Highlights | |
| 16:27:51 | opendevmeet | Removing item from minutes: #topic Sub/related team Highlights | |
| 16:27:51 | bauzas | #undo | |
| 16:28:05 | gibi | the kernel panic happens before we issue detach | |
| 16:28:07 | elodilles | gibi: true | |
| 16:28:21 | gibi | it is either related to the attach or the live migration itself | |
| 16:28:50 | gibi | I have trials placing sleep in different places to see where we are too fast https://review.opendev.org/c/openstack/nova/+/817564 | |
| 16:28:52 | bauzas | which stable branches are impacted ? | |
| 16:28:57 | gibi | stable/victoria | |
| 16:29:01 | bauzas | ubuntu focal-ish I guess ? | |
| 16:29:20 | bauzas | ack thanks | |
| 16:29:21 | elodilles | (and other branches as well, but might be different root causes) | |
| 16:29:45 | gibi | I only see kernel panic in stable/victoria (a lot) and one single failure in stable/wallaby | |
| 16:30:08 | gibi | so if there are detach issues in older stable that is either not causing kernel panic, or we don't see the panic in the logs | |
| 16:30:33 | bauzas | I guess kernel versions are different between branches | |
| 16:30:41 | bauzas | right? | |
| 16:31:06 | gibi | we tested with guest cirros 0.5.1 (victoria default) and 0.5.2 (master default) it is reproducible with both | |
| 16:31:06 | bauzas | ? | |
| 16:31:06 | bauzas | could we imagine somehow to verify another kernel version for stable/victoria | |
| 16:31:23 | bauzas | ack so unrelated | |
| 16:31:31 | gibi | there is a summary here https://bugs.launchpad.net/nova/+bug/1950310/comments/8 | |
| 16:32:19 | bauzas | #link https://bugs.launchpad.net/nova/+bug/1950310/comments/8 explaining the guest kernel panic related to stable/victoria branch | |
| 16:32:22 | sean-k-mooney | ya the fiew cases i looked at with you last week were all happing befoer detach | |
| 16:32:38 | sean-k-mooney | so its either the attach or live migration | |
| 16:32:44 | gibi | sean-k-mooney: I have more logs in the runs of https://review.opendev.org/c/openstack/nova/+/817564 if you are interested | |
| 16:32:52 | sean-k-mooney | i looked downstream at our qemu bugs but didnt see anythign relevent | |
| 16:33:09 | sean-k-mooney | gibi: sure ill try and take a look proably tomorrow | |
| 16:33:15 | sean-k-mooney | but ill open it in a tab | |
| 16:33:57 | gibi | sean-k-mooney: thanks, I will retrigger that patch for a couple times to see if the current sleep before the live migration helps | |
| 16:34:24 | bauzas | a good sleep always helps | |
| 16:34:39 | bauzas | :) | |
| 16:34:45 | elodilles | :] | |
| 16:34:46 | sean-k-mooney | when sleep does not work we can also try a trusty print statement | |
| 16:35:09 | gibi | sleep is not there as a solution but as a troubleshooting to see which step we are too fast :D | |
| 16:35:33 | gibi | and I do have a lot of print(server.console) like statements in the tempest :D | |
| 16:36:10 | sean-k-mooney | i think we can move on but its good you were able to confirm we were attaching before the kerenl finished booting | |
| 16:36:22 | sean-k-mooney | at least in some cases | |
| 16:36:57 | sean-k-mooney | that at least lend weight to the idea we are racing | |
| 16:37:05 | bauzas | ok, let's move on | |
| 16:37:12 | bauzas | again, large agenda todayu | |
| 16:37:12 | gibi | ack | |
| 16:37:17 | bauzas | #topic Sub/related team Highlights | |
| 16:37:23 | bauzas | damn | |
| 16:37:24 | bauzas | #topic Sub/related team Highlights | |
| 16:37:40 | bauzas | Libvirt : lyarwood ? | |
| 16:38:15 | bauzas | I guess nothing to tell | |
| 16:38:19 | bauzas | moving on to the last topic | |
| 16:38:31 | bauzas | #topic Open discussion | |
| 16:39:02 | bauzas | whoami-rajat: please queue | |
| 16:39:09 | whoami-rajat | thanks! | |
| 16:39:19 | bauzas | (kashyapc) Blueprint for review: "Switch to 'virtio' as the default display device" -- https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device | |
| 16:39:27 | bauzas | this is a specless bp ask | |
| 16:39:37 | bauzas | kashyap said " The full rationale is in the blueprint; in short: "cirrus" display device has many limitations and is "considered harmful"[1] by QEMU graphics maintainers since 2014." | |
| 16:40:00 | bauzas | do we need a spec for this bp or are we OK for approving it by now ? | |
| 16:40:19 | whoami-rajat | so lyarwood had a concern with my reimage spec, we agreed to pass the connector info to reimage API (cinder) and cinder will do attachment update and return the connection info with events payload | |
| 16:40:22 | gibi | I think we don't need a spec this is pretty self contained in the libvirt driver | |
| 16:40:23 | bauzas | kashyap was unable to attend the meeting today | |
| 16:40:27 | whoami-rajat | (in PTG) | |
| 16:40:38 | sean-k-mooney | i think we are ok with approving it the main thing to call out is we will be chaing it for existing isntnace too | |
| 16:40:39 | bauzas | whoami-rajat: please hold, sorry | |
| 16:40:43 | whoami-rajat | oh ok | |
| 16:40:45 | gibi | the only open question we had with sean-k-mooney is how to change the default | |
| 16:41:05 | gibi | but kashyap tested it out that changing the default during hard reboot not cause any trouble to guests | |
| 16:41:16 | gibi | as the new video dev has a fallback vga mode | |
| 16:41:17 | bauzas | gibi: I'm thinking hard of any potential upgrade implication | |
| 16:41:32 | sean-k-mooney | right so when we dicussed this before we decied to change it only for new instances to avoid upgrade issue | |
| 16:41:50 | bauzas | correct | |
| 16:41:57 | sean-k-mooney | our downstream QE tested this with windows guests and linux guest and both seamd to be ok with the change | |
| 16:42:01 | bauzas | I'm in favor of not touching the running instances | |
| 16:42:11 | bauzas | or asking to rebuild them | |
| 16:42:18 | gibi | we are not toching the running instance, we only touch hard rebooting instances | |
| 16:42:22 | sean-k-mooney | so kasyap has impletne this for all instnaces | |
| 16:42:49 | bauzas | gibi: which happens when you stop/start, right? | |