| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-12 | |||
| 10:48:01 | lyarwood | I *think* I fsck'd up earlier in the week and forgot to upgrade my tempest.conf to allow volume attached LM | |
| 10:48:09 | lyarwood | update* | |
| 10:50:46 | sean-k-mooney | ah. well in the last 7 days we have hit the detach issue 275 times although my current match seems to trigger 3 times on each event os closer to 90 failed jobs | |
| 10:51:29 | sean-k-mooney | we also seam to have some other libvirt issue too that i have seen intermitently but i cant recall it now | |
| 11:15:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Use initialize APIs to refresh when reported as idempotent https://review.opendev.org/c/openstack/nova/+/720769 | |
| 11:17:47 | lyarwood | elod: https://review.opendev.org/c/openstack/nova/+/780014 - thoughts on this? | |
| 11:24:04 | elod | lyarwood: oh, sorry, I've lost it in my TODOs :S +2+W'd | |
| 11:24:36 | lyarwood | elod: np thanks | |
| 11:31:31 | openstackgerrit | sean mooney proposed openstack/nova stable/train: add functional regression test for bug #1888395 https://review.opendev.org/c/openstack/nova/+/759533 | |
| 11:31:34 | openstack | bug 1888395 in OpenStack Compute (nova) train "live migration of a vm using the single port binding work flow is broken in train as a result of the introduction of sriov live migration" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to Billy Olsen (billy-olsen) | |
| 11:58:47 | stephenfin | functional tests works. hurrah | |
| 11:58:50 | stephenfin | sean-k-mooney: are you addressing gibi's nits and mine or will I? | |
| 11:59:01 | stephenfin | I don't mind. I have to push the functional test anyway | |
| 11:59:14 | sean-k-mooney | i am yes | |
| 11:59:18 | stephenfin | okay, great | |
| 11:59:42 | sean-k-mooney | if you push with -R it wont rebase my stuff and i can cherry pick in your test when i push | |
| 11:59:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | |
| 11:59:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | |
| 11:59:57 | stephenfin | already done ^ | |
| 12:00:04 | sean-k-mooney | yep | |
| 12:00:06 | stephenfin | bauzas: gibi: ^ | |
| 12:00:09 | gibi | ack | |
| 12:34:10 | sean-k-mooney | so artoms socket patch hit the block detach failure again https://review.opendev.org/c/openstack/nova/+/772779 | |
| 12:34:32 | sean-k-mooney | this time in the gate queue | |
| 12:34:53 | sean-k-mooney | shoudl we keep rechecking this or do we have another solution? | |
| 12:35:16 | lyarwood | looks like there are other failures in there as well | |
| 12:35:22 | lyarwood | nova-next failed with a ssh timeout | |
| 12:35:40 | sean-k-mooney | yep but the same tests passed in check | |
| 12:35:50 | lyarwood | nova-ceph-multistore failed with a volume backup failure | |
| 12:35:50 | sean-k-mooney | granted its mixed with other patchs in gate | |
| 12:36:05 | sean-k-mooney | but i dont think those failures are related | |
| 12:36:44 | sean-k-mooney | i think all the failures it hit are intermitent failure in the jobs | |
| 12:36:47 | lyarwood | nova-live-migration failed but not because of a detach issue AFAICT | |
| 12:36:54 | lyarwood | the instance just didn't migrate | |
| 12:37:51 | sean-k-mooney | glanceclient.exc.HTTPNotFound: HTTP 404 Not Found: No image found with ID 202b34e0-db15-437c-8a19-ab391dfcf6e0 | |
| 12:38:44 | sean-k-mooney | although that might be a differnt test? | |
| 12:41:36 | lyarwood | yeah I think so | |
| 12:41:38 | lyarwood | https://zuul.opendev.org/t/openstack/build/98ef805affe54cdd96e32a57b6b87ea1/log/compute1/logs/screen-n-cpu.txt#10011 | |
| 12:41:42 | lyarwood | that's why the LM failed | |
| 12:41:57 | sean-k-mooney | whats annoying about those test failures in general is artoms code changes only take effect if your using pci passthough or sriov which we dont do in any gate jobs. | |
| 12:42:02 | lyarwood | but why is that a different request-id | |
| 12:43:21 | sean-k-mooney | libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-vexxhost-ca-ymq-1-0023456556:49152': Connection refused | |
| 12:44:30 | sean-k-mooney | tempest-LiveMigrationTest-1563426117 tempest-LiveMigrationTest-1563426117-project-admin] Could not generate host nqn: [Errno 2] No such file or directory | |
| 12:44:52 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/98ef805affe54cdd96e32a57b6b87ea1/log/compute1/logs/screen-n-cpu.txt#10481 | |
| 12:45:07 | sean-k-mooney | nvme gen-hostnqn | tee /etc/nvme/hostnqn | |
| 12:45:27 | sean-k-mooney | that odd | |
| 12:45:28 | lyarwood | yeah I think that's ignored | |
| 12:45:34 | lyarwood | it's the os-brick connector | |
| 12:45:37 | sean-k-mooney | why are we doing nvme things | |
| 12:46:15 | sean-k-mooney | ya i think its ignored too but the gate cant test nvme as far as im aware | |
| 12:46:31 | sean-k-mooney | so not sure why the nvmeof connector is trying to do anything | |
| 12:46:41 | lyarwood | yeah the os-brick connector doesn't know ahead of time what the actual volume type is | |
| 12:47:11 | lyarwood | so it just gathers info on all of the possible IQN/HBAs etc it can | |
| 12:47:26 | sean-k-mooney | thats not ideal but ok | |
| 12:47:35 | lyarwood | yeah it's a really old wrinkle in the APIs | |
| 12:47:58 | lyarwood | ideally you'd want cinder API to tell the caller what the type is first and what connector info it needs to map the volume | |
| 12:48:07 | sean-k-mooney | artom recheked this so im going to go back to fixing nits | |
| 12:48:11 | sean-k-mooney | lyarwood: yep | |
| 12:48:23 | sean-k-mooney | like with os-vif | |
| 12:48:40 | sean-k-mooney | we use the vif-type and vnic-type to only call the correct plugin | |
| 12:48:51 | sean-k-mooney | we never do any kind of probing | |
| 13:11:33 | sean-k-mooney | stephenfin: by the way a possible gotch ya with pre-commit | |
| 13:12:03 | sean-k-mooney | i dont think it runs if you do a git add during an interactive rebase and then do git rebase --continue | |
| 13:12:19 | sean-k-mooney | you have to do git commit --amend first | |
| 13:14:34 | bauzas | sean-k-mooney: you got a merge conflict ? | |
| 13:14:53 | sean-k-mooney | for? | |
| 13:16:30 | sean-k-mooney | im interactive rebaseing to adress the fact i removed the devpath in the wong commit and im adress all the style nits in the rest of the following patchs since they are going to be rebased as a result anyway | |
| 13:16:57 | sean-k-mooney | with precommit if you just to rebase --continue it just does not run the commit hook i think | |
| 13:17:05 | sean-k-mooney | since it technially not a commit | |
| 13:17:36 | sean-k-mooney | so if you want precommit to run when doing interactive rebases and your editing files then make sure to comit before moving on | |
| 13:19:16 | sean-k-mooney | bauzas: if you are asking will there be a merge conflict then yes | |
| 13:19:36 | sean-k-mooney | a very minor one with the pci socket policy patch | |
| 13:19:55 | sean-k-mooney | its a one line conflict in a test | |
| 13:21:04 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add vDPA nodedev parsing https://review.opendev.org/c/openstack/nova/+/770533 | |
| 13:21:05 | openstackgerrit | sean mooney proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | |
| 13:21:05 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add guest generation for vDPA https://review.opendev.org/c/openstack/nova/+/770532 | |
| 13:21:06 | openstackgerrit | sean mooney proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | |
| 13:21:07 | openstackgerrit | sean mooney proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | |
| 13:21:36 | sean-k-mooney | stephenfin: that should adress all the nits too | |
| 13:22:50 | lyarwood | I can't recall ever seeing this written down anywhere but do we have a policy against passing context around, if only to improve logging by ensuring a request-id is logged alongside each message? | |
| 13:23:15 | lyarwood | I'm trying to debug the volume detach failure and am getting pissed with the lack of context in some places | |
| 13:23:34 | lyarwood | I've added instance=instance to all of the LOG calls in the volume drivers within libvirt now | |
| 13:23:41 | sean-k-mooney | i personally try not to pass it if i can avoid it | |
| 13:23:46 | lyarwood | but as we don't pass context down into these there's no request-id | |
| 13:23:54 | sean-k-mooney | but i almost never use request id | |
| 13:24:06 | lyarwood | yeah, I'm a heavy heavy user of it now | |
| 13:24:18 | lyarwood | and while I get that it's pointless if it isn't used | |
| 13:24:27 | lyarwood | the request-id is soooooooo useful | |
| 13:24:51 | sean-k-mooney | if you have the instance uuid that normally enough | |
| 13:24:57 | sean-k-mooney | it can be yes | |
| 13:25:24 | lyarwood | you just end up making assumptions I find | |
| 13:25:25 | sean-k-mooney | but unless we can come up with a better way then pass it to every function i would prefer not to | |
| 13:25:51 | lyarwood | I'll have to read the oslo.log code again to recall what it actually looks for | |
| 13:25:58 | artom | lyarwood, I thought the request ID was stashed in a thread global or something? At least that's what I was told... | |
| 13:25:58 | lyarwood | it might look for other things aside from context | |
| 13:26:19 | sean-k-mooney | artom: i dont think so | |
| 13:26:20 | lyarwood | hmm I thought oslo.log pulled it out of context tbh | |
| 13:26:27 | sean-k-mooney | that would not work well with eventlests | |
| 13:26:51 | artom | *shrug* I have to do the dad taxi thing anyways | |
| 13:26:55 | sean-k-mooney | unless you mean a green thread local variable of some kind | |