| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-12 | |||
| 12:36:44 | sean-k-mooney | i think all the failures it hit are intermitent failure in the jobs | |
| 12:36:47 | lyarwood | nova-live-migration failed but not because of a detach issue AFAICT | |
| 12:36:54 | lyarwood | the instance just didn't migrate | |
| 12:37:51 | sean-k-mooney | glanceclient.exc.HTTPNotFound: HTTP 404 Not Found: No image found with ID 202b34e0-db15-437c-8a19-ab391dfcf6e0 | |
| 12:38:44 | sean-k-mooney | although that might be a differnt test? | |
| 12:41:36 | lyarwood | yeah I think so | |
| 12:41:38 | lyarwood | https://zuul.opendev.org/t/openstack/build/98ef805affe54cdd96e32a57b6b87ea1/log/compute1/logs/screen-n-cpu.txt#10011 | |
| 12:41:42 | lyarwood | that's why the LM failed | |
| 12:41:57 | sean-k-mooney | whats annoying about those test failures in general is artoms code changes only take effect if your using pci passthough or sriov which we dont do in any gate jobs. | |
| 12:42:02 | lyarwood | but why is that a different request-id | |
| 12:43:21 | sean-k-mooney | libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-vexxhost-ca-ymq-1-0023456556:49152': Connection refused | |
| 12:44:30 | sean-k-mooney | tempest-LiveMigrationTest-1563426117 tempest-LiveMigrationTest-1563426117-project-admin] Could not generate host nqn: [Errno 2] No such file or directory | |
| 12:44:52 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/98ef805affe54cdd96e32a57b6b87ea1/log/compute1/logs/screen-n-cpu.txt#10481 | |
| 12:45:07 | sean-k-mooney | nvme gen-hostnqn | tee /etc/nvme/hostnqn | |
| 12:45:27 | sean-k-mooney | that odd | |
| 12:45:28 | lyarwood | yeah I think that's ignored | |
| 12:45:34 | lyarwood | it's the os-brick connector | |
| 12:45:37 | sean-k-mooney | why are we doing nvme things | |
| 12:46:15 | sean-k-mooney | ya i think its ignored too but the gate cant test nvme as far as im aware | |
| 12:46:31 | sean-k-mooney | so not sure why the nvmeof connector is trying to do anything | |
| 12:46:41 | lyarwood | yeah the os-brick connector doesn't know ahead of time what the actual volume type is | |
| 12:47:11 | lyarwood | so it just gathers info on all of the possible IQN/HBAs etc it can | |
| 12:47:26 | sean-k-mooney | thats not ideal but ok | |
| 12:47:35 | lyarwood | yeah it's a really old wrinkle in the APIs | |
| 12:47:58 | lyarwood | ideally you'd want cinder API to tell the caller what the type is first and what connector info it needs to map the volume | |
| 12:48:07 | sean-k-mooney | artom recheked this so im going to go back to fixing nits | |
| 12:48:11 | sean-k-mooney | lyarwood: yep | |
| 12:48:23 | sean-k-mooney | like with os-vif | |
| 12:48:40 | sean-k-mooney | we use the vif-type and vnic-type to only call the correct plugin | |
| 12:48:51 | sean-k-mooney | we never do any kind of probing | |
| 13:11:33 | sean-k-mooney | stephenfin: by the way a possible gotch ya with pre-commit | |
| 13:12:03 | sean-k-mooney | i dont think it runs if you do a git add during an interactive rebase and then do git rebase --continue | |
| 13:12:19 | sean-k-mooney | you have to do git commit --amend first | |
| 13:14:34 | bauzas | sean-k-mooney: you got a merge conflict ? | |
| 13:14:53 | sean-k-mooney | for? | |
| 13:16:30 | sean-k-mooney | im interactive rebaseing to adress the fact i removed the devpath in the wong commit and im adress all the style nits in the rest of the following patchs since they are going to be rebased as a result anyway | |
| 13:16:57 | sean-k-mooney | with precommit if you just to rebase --continue it just does not run the commit hook i think | |
| 13:17:05 | sean-k-mooney | since it technially not a commit | |
| 13:17:36 | sean-k-mooney | so if you want precommit to run when doing interactive rebases and your editing files then make sure to comit before moving on | |
| 13:19:16 | sean-k-mooney | bauzas: if you are asking will there be a merge conflict then yes | |
| 13:19:36 | sean-k-mooney | a very minor one with the pci socket policy patch | |
| 13:19:55 | sean-k-mooney | its a one line conflict in a test | |
| 13:21:04 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add vDPA nodedev parsing https://review.opendev.org/c/openstack/nova/+/770533 | |
| 13:21:05 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add guest generation for vDPA https://review.opendev.org/c/openstack/nova/+/770532 | |
| 13:21:05 | openstackgerrit | sean mooney proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | |
| 13:21:06 | openstackgerrit | sean mooney proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | |
| 13:21:07 | openstackgerrit | sean mooney proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | |
| 13:21:36 | sean-k-mooney | stephenfin: that should adress all the nits too | |
| 13:22:50 | lyarwood | I can't recall ever seeing this written down anywhere but do we have a policy against passing context around, if only to improve logging by ensuring a request-id is logged alongside each message? | |
| 13:23:15 | lyarwood | I'm trying to debug the volume detach failure and am getting pissed with the lack of context in some places | |
| 13:23:34 | lyarwood | I've added instance=instance to all of the LOG calls in the volume drivers within libvirt now | |
| 13:23:41 | sean-k-mooney | i personally try not to pass it if i can avoid it | |
| 13:23:46 | lyarwood | but as we don't pass context down into these there's no request-id | |
| 13:23:54 | sean-k-mooney | but i almost never use request id | |
| 13:24:06 | lyarwood | yeah, I'm a heavy heavy user of it now | |
| 13:24:18 | lyarwood | and while I get that it's pointless if it isn't used | |
| 13:24:27 | lyarwood | the request-id is soooooooo useful | |
| 13:24:51 | sean-k-mooney | if you have the instance uuid that normally enough | |
| 13:24:57 | sean-k-mooney | it can be yes | |
| 13:25:24 | lyarwood | you just end up making assumptions I find | |
| 13:25:25 | sean-k-mooney | but unless we can come up with a better way then pass it to every function i would prefer not to | |
| 13:25:51 | lyarwood | I'll have to read the oslo.log code again to recall what it actually looks for | |
| 13:25:58 | lyarwood | it might look for other things aside from context | |
| 13:25:58 | artom | lyarwood, I thought the request ID was stashed in a thread global or something? At least that's what I was told... | |
| 13:26:19 | sean-k-mooney | artom: i dont think so | |
| 13:26:20 | lyarwood | hmm I thought oslo.log pulled it out of context tbh | |
| 13:26:27 | sean-k-mooney | that would not work well with eventlests | |
| 13:26:51 | artom | *shrug* I have to do the dad taxi thing anyways | |
| 13:26:55 | sean-k-mooney | unless you mean a green thread local variable of some kind | |
| 13:27:10 | sean-k-mooney | bu it i still would assume its not form that | |
| 13:27:21 | lyarwood | oh it's both | |
| 13:27:30 | lyarwood | https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/formatters.py#L61-L78 | |
| 13:29:10 | lyarwood | ah right cool | |
| 13:29:19 | lyarwood | okay so passing context isn't required | |
| 13:30:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Ensure all volume drivers log the instance whenever possible https://review.opendev.org/c/openstack/nova/+/780260 | |
| 13:30:59 | sean-k-mooney | https://github.com/openstack/oslo.context/blob/0d02866365bc8b779aef9ebd0b79a52c96ae40e5/oslo_context/context.py#L508-L513 | |
| 13:31:08 | sean-k-mooney | so it can retrun None | |
| 13:31:34 | sean-k-mooney | _request_store is a thread local | |
| 13:31:36 | sean-k-mooney | https://github.com/openstack/oslo.context/blob/0d02866365bc8b779aef9ebd0b79a52c96ae40e5/oslo_context/context.py#L40 | |
| 13:32:10 | sean-k-mooney | so if the green thread resoumes on a different thread becaue we are usign a thread pool it might be empty | |
| 13:32:36 | lyarwood | kk makes sense, both connect and disconnect wait behind a lock | |
| 13:32:45 | lyarwood | so I assume that's where we end up losing it | |
| 13:32:49 | lyarwood | ew | |
| 13:33:07 | sean-k-mooney | do i want to know what you just saw | |
| 13:33:37 | lyarwood | that was more at the logging changing because of thread local storage | |
| 13:34:03 | lyarwood | I'm looking at yet another example failure of the detach thing | |
| 13:34:11 | lyarwood | another test attaches a volume | |
| 13:34:19 | lyarwood | the failing test attaches a volume | |
| 13:34:23 | lyarwood | the other test detaches | |
| 13:34:31 | lyarwood | and then our failing test attempts to detach | |
| 13:34:45 | sean-k-mooney | thats also ew :) | |
| 13:35:08 | lyarwood | well it's all locked and serialised so that's fine | |
| 13:35:18 | lyarwood | but the failing test just times out trying to detach the device | |
| 13:35:19 | sean-k-mooney | can we just chnage this locally in nova to workaround the issue | |
| 13:35:22 | lyarwood | there's some stuff in syslog | |
| 13:35:33 | lyarwood | but I can't get my head around it tbh | |
| 13:36:16 | sean-k-mooney | you still think this is due to conflicting WWNs right | |
| 13:37:14 | lyarwood | yeah I don't think that's helping but I can't find a smoking gun here | |
| 13:47:17 | belmoreira | hi, we are live migrating thousands of instances because hardware retirement and hit an old issue :) this was discussed in the past for instance creation, but now I'm facing it at live migration, when the migration allocation is created. https://bugs.launchpad.net/nova/+bug/1918419 | |
| 13:47:19 | openstack | Launchpad bug 1918419 in OpenStack Compute (nova) "vCPU resource max_unit is hardcoded" [Undecided,New] | |