| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-22 | |||
| 10:47:17 | bauzas | sean-k-mooney: I see then | |
| 10:47:20 | sean-k-mooney[m] | on the plus side whitebox caught htis | |
| 10:47:47 | bauzas | when is the target guest defined and paused ? | |
| 10:47:50 | sean-k-mooney[m] | on the down side we are not yet running whitebox on nova | |
| 10:48:00 | bauzas | I thought it was when we were calling qemu migrate | |
| 10:48:06 | gibi | would be nice to get bug reports i can link to. | |
| 10:48:22 | sean-k-mooney[m] | bauzas: its defiend and paused by libvirt when we call migrateToUri3 | |
| 10:48:44 | bauzas | which is not done in pre-livemigrate | |
| 10:48:52 | sean-k-mooney[m] | no | |
| 10:48:54 | bauzas | but I see the reason on turning on the core before | |
| 10:49:00 | bauzas | then in premigrate | |
| 10:49:02 | bauzas | gotcha | |
| 10:49:03 | sean-k-mooney[m] | its done after pre-livemigate | |
| 10:49:28 | sean-k-mooney[m] | but its the call to libvirt on the source host that cause libvirt to create the vm in the paused sate on the dest | |
| 10:49:48 | bauzas | yeah, same point than for the mdev things we discussed :) | |
| 10:50:06 | bauzas | bad news is that internals of live-migration was blind for me in Antelope timle | |
| 10:50:11 | bauzas | now, this is no longer the case | |
| 10:50:13 | sean-k-mooney[m] | yes its the same code paths as the mdevs | |
| 10:50:37 | gibi | https://github.com/openstack-k8s-operators/nova-operator/pull/702 revert is up | |
| 10:51:02 | sean-k-mooney[m] | by the way nova-next is blocked | |
| 10:51:09 | sean-k-mooney[m] | im going to fix that shortly | |
| 10:51:26 | sean-k-mooney[m] | we are messing with the port bidnign profile for some reason in the post hook | |
| 10:51:29 | sean-k-mooney[m] | we should not be doing that | |
| 10:54:15 | sean-k-mooney[m] | gibi: approved. im going to grab coffee and ill work on unblocking the gate when i get back | |
| 10:54:48 | opendevreview | Doug Szumski proposed openstack/nova master: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/909122 | |
| 11:04:57 | bauzas | sean-k-mooney: gibi: we could add a note in our upstream doc saying we don't support live-migration yet due to a bug | |
| 11:19:03 | gibi | I would rather fix the bugs instead | |
| 11:21:25 | sean-k-mooney | bauzas: same im not ok with doc fixes like that in general i would prefer an api block or an actual fix | |
| 11:21:47 | bauzas | cool then | |
| 11:22:16 | sean-k-mooney | bauzas: artom has started workign on a fix | |
| 11:22:26 | bauzas | gtk | |
| 11:22:38 | sean-k-mooney | so lets just help him do that and we can file an upstream bug to track it properly | |
| 12:11:08 | opendevreview | sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 12:11:18 | sean-k-mooney | bauzas: gibi ^ that will fix the gate blocker | |
| 12:29:39 | opendevreview | Merged openstack/os-vif master: Drop wrong stacklevel https://review.opendev.org/c/openstack/os-vif/+/909682 | |
| 12:30:45 | gibi | +2 with some followup thinking | |
| 12:31:56 | artom | gibi, sean-k-mooney, there's still something weird going on, https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/909785 isn't fully passing | |
| 12:32:14 | artom | It's _better_ with the two POC patches and Uggla's cpu_shared_set live migration patches, but still red | |
| 12:55:21 | sean-k-mooney | gibi: we can likely refactor this to use python to set the extra atributes on the port | |
| 12:55:29 | sean-k-mooney | just not right now | |
| 12:56:21 | sean-k-mooney | we can however just drop a python file into the hooks dir that import novas service user stuff and uses nova's code to set the my_key data | |
| 13:04:57 | gibi | sean-k-mooney: ack, i definitely wont block on this | |
| 13:07:52 | sean-k-mooney | i belvie we have unit tests that make sure we dont clobber but i agree the integration test was nice ot have | |
| 13:08:20 | gibi | artom: Feb 22 03:45:17.085381 np0036833327 nova-compute[103912]: ERROR oslo_messaging.rpc.server NotImplementedError: Cannot load 'emulator_pins' in the base class | |
| 13:08:53 | sean-k-mooney | ya so that depending on a patch that is not in the patch chain i belive | |
| 13:09:11 | sean-k-mooney | gibi: i think the whitebox job is using two depend on again nova | |
| 13:09:51 | sean-k-mooney | https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/909785 | |
| 13:09:53 | sean-k-mooney | yep | |
| 13:10:33 | sean-k-mooney | or not | |
| 13:10:47 | gibi | it seem the numa_info has no emulator_pin field | |
| 13:10:51 | sean-k-mooney | i tought it might be in https://review.opendev.org/c/openstack/nova/+/877773/8 | |
| 13:11:36 | gibi | a functional test in that patch probably can reproduce this issue | |
| 13:11:44 | sean-k-mooney | the emulator thread are ment to be included in cpu_pins | |
| 13:12:55 | sean-k-mooney | gibi: one of the thing i disucssed with artom was https://github.com/openstack/nova/blob/master/nova/objects/instance_numa.py#L291-L296 | |
| 13:13:04 | sean-k-mooney | is ment to have the emulator pinning in it too | |
| 13:13:51 | sean-k-mooney | actully looking at https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L144 | |
| 13:14:02 | sean-k-mooney | emulator_pins is in the object | |
| 13:14:10 | sean-k-mooney | so artom is just missing an s | |
| 13:14:18 | sean-k-mooney | since it sa set since the a range | |
| 13:16:09 | sean-k-mooney | hum no the patch is correct which means its not populated? | |
| 13:16:47 | sean-k-mooney | https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/libvirt/migration.py#L113-L117 | |
| 13:17:35 | sean-k-mooney | @artom i mostly aggre with this comment but i belive we dont alwasy set emulatorpin in the xml | |
| 13:17:52 | sean-k-mooney | https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/libvirt/migration.py#L113-L126 | |
| 13:18:05 | artom | Yeah, I think I need an if there | |
| 13:18:15 | artom | It won't be set if there's no emulator thread policy | |
| 13:18:36 | sean-k-mooney | if emulatorpin is not found you need to default it to vcpupin | |
| 13:19:08 | sean-k-mooney | so ya you need an if here https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/libvirt/migration.py#L127 | |
| 13:19:27 | sean-k-mooney | well | |
| 13:19:33 | sean-k-mooney | actully proably not there | |
| 13:19:45 | sean-k-mooney | but where we buidl teh dst_numa_info in the first place | |
| 13:19:59 | sean-k-mooney | well either works i guess | |
| 13:20:39 | sean-k-mooney | for backport reason we proably want to have _update_numa_xml have the fallback if its not set in dst_numa_info | |
| 13:20:52 | sean-k-mooney | but we should fix where we are creating the numa info to also set it properly | |
| 13:22:02 | sean-k-mooney | so add somethign like | |
| 13:23:18 | sean-k-mooney | emulator_pin = info.emulator_pins if info.emulator_pins else info.cpu_pins.values() | |
| 13:23:38 | sean-k-mooney | emulatorpin.set('cpuset',hardware.format_cpu_spec(emulator_pin)) | |
| 13:42:44 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 14:53:39 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Auto set heartbeat_in_pthread for wsgi services https://review.opendev.org/c/openstack/nova/+/909880 | |
| 15:31:48 | mnaser | It seems `nova-grenade-multinode` is broken in `stable/zed`. Does anyone know of anything that sticks out | |
| 15:32:13 | mnaser | https://review.opendev.org/c/openstack/nova/+/909098 we're eating up resources with folks rechecking it, we're at 3x, i think the job is deffo broken | |
| 15:33:13 | mnaser | ok, that's neutron being broken | |
| 15:33:19 | mnaser | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_39b/909098/2/check/nova-grenade-multinode/39bc664/controller/logs/screen-q-svc.txt | |
| 15:34:05 | frickler | iiuc grenade jobs should be dropped from zed now that yoga is unmaintained? | |
| 15:43:15 | dansmith | idk that we actually said that (removing grenade if the source is unmaintained) but that make sense to me | |
| 15:43:58 | clarkb | historically we've dropped grenade jobs for N+1 when N is no longer maintained | |
| 15:44:10 | clarkb | beacuse the grenade jobs start at N and upgrade to N+1 and you very quickly bitrot | |
| 15:46:01 | dansmith | you mean historically...removed right? I meant in response to the recent policy to have branches that are specifically _unmaintained_ but still around | |
| 15:46:11 | clarkb | correct historically | |
| 15:47:05 | dansmith | in the past, IIRC, we'd remove a grenade when it upgrades from a branch that gets deleted because we can't land a fix to make it installable, but it's slightly less clear when the branch exists but is in the newly-minted unmaintained state | |
| 15:47:25 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: power up dedicated cores during pre_live_migration https://review.opendev.org/c/openstack/nova/+/909806 | |
| 15:48:05 | artom | Let's see if the emulator_pins fix is enough | |
| 15:53:15 | frickler | those grenade runs seem also have been in a weird situation where the stable/yoga branches were already removed in the repos, but still present in our in-image-git-cache. but I also don't understand why n-cond seems to get started, but doesn't produce any log | |
| 15:57:34 | melwitt | I dunno if yall have seen but it looks like nova-next is at a 100% fail rate, POST_FAILURE | |
| 15:57:41 | melwitt | ForbiddenException: 403: Client Error for url: https://213.32.74.123:9696/networking/v2.0/ports/f5b273ab-1c98-44f6-9664-8c1a16d5abda, (rule:update_port and rule:update_port:binding:profile) is disallowed by policy | |
| 15:57:41 | melwitt | + openstack port unset --binding-profile allocation port-normal-qos | |
| 15:57:41 | melwitt | Deleting allocation key from the binding:profile of the bandwidth aware port | |
| 15:58:36 | melwitt | not sure what changed that could have affected policy. I'll look around | |
| 16:02:18 | sean-k-mooney | melwitt: i have a fix for it | |
| 16:02:40 | melwitt | ok thank goodness | |
| 16:02:43 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/909859 | |