| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-16 | |||
| 16:28:21 | gibi | it is either related to the attach or the live migration itself | |
| 16:28:50 | gibi | I have trials placing sleep in different places to see where we are too fast https://review.opendev.org/c/openstack/nova/+/817564 | |
| 16:28:52 | bauzas | which stable branches are impacted ? | |
| 16:28:57 | gibi | stable/victoria | |
| 16:29:01 | bauzas | ubuntu focal-ish I guess ? | |
| 16:29:20 | bauzas | ack thanks | |
| 16:29:21 | elodilles | (and other branches as well, but might be different root causes) | |
| 16:29:45 | gibi | I only see kernel panic in stable/victoria (a lot) and one single failure in stable/wallaby | |
| 16:30:08 | gibi | so if there are detach issues in older stable that is either not causing kernel panic, or we don't see the panic in the logs | |
| 16:30:33 | bauzas | I guess kernel versions are different between branches | |
| 16:30:41 | bauzas | right? | |
| 16:31:06 | gibi | we tested with guest cirros 0.5.1 (victoria default) and 0.5.2 (master default) it is reproducible with both | |
| 16:31:06 | bauzas | ? | |
| 16:31:06 | bauzas | could we imagine somehow to verify another kernel version for stable/victoria | |
| 16:31:23 | bauzas | ack so unrelated | |
| 16:31:31 | gibi | there is a summary here https://bugs.launchpad.net/nova/+bug/1950310/comments/8 | |
| 16:32:19 | bauzas | #link https://bugs.launchpad.net/nova/+bug/1950310/comments/8 explaining the guest kernel panic related to stable/victoria branch | |
| 16:32:22 | sean-k-mooney | ya the fiew cases i looked at with you last week were all happing befoer detach | |
| 16:32:38 | sean-k-mooney | so its either the attach or live migration | |
| 16:32:44 | gibi | sean-k-mooney: I have more logs in the runs of https://review.opendev.org/c/openstack/nova/+/817564 if you are interested | |
| 16:32:52 | sean-k-mooney | i looked downstream at our qemu bugs but didnt see anythign relevent | |
| 16:33:09 | sean-k-mooney | gibi: sure ill try and take a look proably tomorrow | |
| 16:33:15 | sean-k-mooney | but ill open it in a tab | |
| 16:33:57 | gibi | sean-k-mooney: thanks, I will retrigger that patch for a couple times to see if the current sleep before the live migration helps | |
| 16:34:24 | bauzas | a good sleep always helps | |
| 16:34:39 | bauzas | :) | |
| 16:34:45 | elodilles | :] | |
| 16:34:46 | sean-k-mooney | when sleep does not work we can also try a trusty print statement | |
| 16:35:09 | gibi | sleep is not there as a solution but as a troubleshooting to see which step we are too fast :D | |
| 16:35:33 | gibi | and I do have a lot of print(server.console) like statements in the tempest :D | |
| 16:36:10 | sean-k-mooney | i think we can move on but its good you were able to confirm we were attaching before the kerenl finished booting | |
| 16:36:22 | sean-k-mooney | at least in some cases | |
| 16:36:57 | sean-k-mooney | that at least lend weight to the idea we are racing | |
| 16:37:05 | bauzas | ok, let's move on | |
| 16:37:12 | bauzas | again, large agenda todayu | |
| 16:37:12 | gibi | ack | |
| 16:37:17 | bauzas | #topic Sub/related team Highlights | |
| 16:37:23 | bauzas | damn | |
| 16:37:24 | bauzas | #topic Sub/related team Highlights | |
| 16:37:40 | bauzas | Libvirt : lyarwood ? | |
| 16:38:15 | bauzas | I guess nothing to tell | |
| 16:38:19 | bauzas | moving on to the last topic | |
| 16:38:31 | bauzas | #topic Open discussion | |
| 16:39:02 | bauzas | whoami-rajat: please queue | |
| 16:39:09 | whoami-rajat | thanks! | |
| 16:39:19 | bauzas | (kashyapc) Blueprint for review: "Switch to 'virtio' as the default display device" -- https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device | |
| 16:39:27 | bauzas | this is a specless bp ask | |
| 16:39:37 | bauzas | kashyap said " The full rationale is in the blueprint; in short: "cirrus" display device has many limitations and is "considered harmful"[1] by QEMU graphics maintainers since 2014." | |
| 16:40:00 | bauzas | do we need a spec for this bp or are we OK for approving it by now ? | |
| 16:40:19 | whoami-rajat | so lyarwood had a concern with my reimage spec, we agreed to pass the connector info to reimage API (cinder) and cinder will do attachment update and return the connection info with events payload | |
| 16:40:22 | gibi | I think we don't need a spec this is pretty self contained in the libvirt driver | |
| 16:40:23 | bauzas | kashyap was unable to attend the meeting today | |
| 16:40:27 | whoami-rajat | (in PTG) | |
| 16:40:38 | sean-k-mooney | i think we are ok with approving it the main thing to call out is we will be chaing it for existing isntnace too | |
| 16:40:39 | bauzas | whoami-rajat: please hold, sorry | |
| 16:40:43 | whoami-rajat | oh ok | |
| 16:40:45 | gibi | the only open question we had with sean-k-mooney is how to change the default | |
| 16:41:05 | gibi | but kashyap tested it out that changing the default during hard reboot not cause any trouble to guests | |
| 16:41:16 | gibi | as the new video dev has a fallback vga mode | |
| 16:41:17 | bauzas | gibi: I'm thinking hard of any potential upgrade implication | |
| 16:41:32 | sean-k-mooney | right so when we dicussed this before we decied to change it only for new instances to avoid upgrade issue | |
| 16:41:50 | bauzas | correct | |
| 16:41:57 | sean-k-mooney | our downstream QE tested this with windows guests and linux guest and both seamd to be ok with the change | |
| 16:42:01 | bauzas | I'm in favor of not touching the running instances | |
| 16:42:11 | bauzas | or asking to rebuild them | |
| 16:42:18 | gibi | we are not toching the running instance, we only touch hard rebooting instances | |
| 16:42:22 | sean-k-mooney | so kasyap has impletne this for all instnaces | |
| 16:42:49 | bauzas | gibi: which happens when you stop/start, right? | |
| 16:42:54 | sean-k-mooney | bauzas: yes as gibi says it will only take effect when the xml is next regenreted | |
| 16:42:54 | gibi | right | |
| 16:42:59 | gibi | it happens while the guest is not running | |
| 16:43:21 | gibi | it is not an unplug/plug for a running guest | |
| 16:43:29 | bauzas | do we want admins to opt-in instances ? | |
| 16:43:41 | bauzas | or do we agree it would be done automatically? | |
| 16:43:42 | sean-k-mooney | it will happen on start/stop hard reboot or a non live move operations | |
| 16:44:03 | gibi | bauzas: I trust kashyap that it is safe to change this device | |
| 16:44:14 | bauzas | do we also want to have a nova-status upgrade check for yoga about this ? | |
| 16:44:23 | sean-k-mooney | no | |
| 16:44:24 | bauzas | gibi: me too | |
| 16:44:29 | sean-k-mooney | why would we need too | |
| 16:44:35 | sean-k-mooney | we are not removing support for cirrus | |
| 16:44:36 | gibi | we don't remove cirros | |
| 16:44:41 | sean-k-mooney | jsut not the default | |
| 16:44:46 | gibi | yepp | |
| 16:45:00 | sean-k-mooney | gibi: context is downstream it is being remvoed form rhel 9 | |
| 16:45:04 | bauzas | sean-k-mooney: sure, that just means that long-living instances could continue running cirros | |
| 16:45:14 | sean-k-mooney | so wwe need to care about it for our product | |
| 16:45:24 | sean-k-mooney | actully cirrus is not beeing remvoed in rhel 9 | |
| 16:45:35 | sean-k-mooney | but like in rhel 10 | |
| 16:46:06 | sean-k-mooney | bauzas: yep which i think is ok | |
| 16:46:27 | sean-k-mooney | we coudl have a nova status check but it woudl have to run on the compute nodes | |
| 16:46:35 | sean-k-mooney | which is kind of not nice | |
| 16:46:40 | sean-k-mooney | since it woudl have to check the xmls | |
| 16:46:51 | bauzas | I know | |
| 16:46:53 | sean-k-mooney | so i woudl not add it personally | |
| 16:47:19 | bauzas | I'm just saying that we enter a time that could last long | |
| 16:47:20 | gibi | I agree, we don't need upgrade check | |
| 16:48:16 | sean-k-mooney | shal we continue this in the patch review | |
| 16:48:17 | bauzas | but agreed on the fact this is not a problem until cirros support is removed and this is not an upstream question | |
| 16:48:31 | bauzas | sean-k-mooney: you're right, nothing needing a spec | |