| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-06 | |||
| 19:23:11 | mriedem | i'm seeing this during a rebuild | |
| 19:23:13 | mriedem | where we're not migrating | |
| 19:23:21 | mriedem | but the rebuild code also handles evacuate, which is the migration | |
| 19:23:42 | mriedem | but it loos like we end up lazy-loading numa_topology, pci_requests and pci_devices one after another just to ultimately do nothing | |
| 19:24:48 | dansmith | yeah I guess not.. you see that debug log warning that there's no context too I guess? | |
| 19:25:27 | mriedem | http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz#_Jun_06_13_14_02_547424 | |
| 19:25:31 | mriedem | yup | |
| 19:25:32 | cfriesen_ | mriedem: if you're not actually migrating I think it'd be safe to bypass the mutated_migration_context | |
| 19:25:35 | mriedem | see the next 4 there | |
| 19:25:45 | mriedem | it lazy-loads 3 fields, and then says "you're dumb" | |
| 19:26:19 | mriedem | and i believe each lazy-load on those fields is going to be a full pull from the db via conductor... | |
| 19:27:58 | kevinbenton | mriedem: in a rebuild what does nova tell neutron? | |
| 19:28:06 | kevinbenton | mriedem: because the host_id doesn't change, right? | |
| 19:28:48 | cfriesen_ | mriedem: can a rebuild end up picking a new host? | |
| 19:29:46 | cfriesen_ | (when we're not doing an evacuate, I mean) | |
| 19:30:23 | mriedem | cfriesen_: kevinbenton: no, same host | |
| 19:30:35 | cfriesen_ | mriedem:, right, we specify the host in that case | |
| 19:30:39 | mriedem | kevinbenton: so my theory is neutron is not sending the event maybe because the host doesn't change | |
| 19:30:57 | mriedem | but i'd think that would be 100% fail | |
| 19:31:28 | mriedem | so i wonder if we're hitting a race where neutron is queueing notifications and thinks it already told nova something about a given port | |
| 19:32:29 | kevinbenton | mriedem: we did have a recent change on the neutron side that reduced some port status changes when they weren't needed | |
| 19:32:58 | kevinbenton | mriedem: so it's possible that there is a race that was being covered by side effects from that before | |
| 19:33:35 | kevinbenton | mriedem: do you have a set of logs from one of these failures? | |
| 19:35:25 | mriedem | kevinbenton: yeah that bug linked above has a log dive from me | |
| 19:35:30 | mriedem | showing where things happen on the nova side | |
| 19:37:43 | efried | sdague Can you please help me understand whether/how global_request_id affordance is needed in https://review.openstack.org/#/c/412634/28/nova/image/glance.py ? | |
| 19:38:25 | efried | I'm kinda hoping it's implicitly part of the ksa session... | |
| 19:38:45 | efried | ...or auth | |
| 19:40:38 | efried | ...because of the context | |
| 19:49:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | |
| 19:49:50 | openstackgerrit | Eric Fried proposed openstack/nova master: Add service_token for nova-glance interaction https://review.openstack.org/419662 | |
| 19:51:26 | sdague | efried: it's not implicit | |
| 19:52:08 | efried | sdague So I need to add the header explicitly? | |
| 19:52:08 | sdague | efried: it's a single parameter to pass | |
| 19:52:17 | sdague | one sec | |
| 19:52:53 | sdague | efried: https://review.openstack.org/#/c/467242/6/nova/image/glance.py | |
| 19:53:10 | efried | sdague Right, I'm rebasing against that. | |
| 19:53:18 | efried | sdague But in the new version, we don't do any of that stuff. | |
| 19:53:32 | sdague | efried: yeh, so just add global_request_id=context.global_id | |
| 19:53:50 | sdague | to the constructor args for glanceclient.Client | |
| 19:54:23 | efried | sdague Okay, will do. Thanks. | |
| 19:54:43 | sdague | like this - https://github.com/openstack/nova/blob/2f712b84ecf8d149833dea17d2fa4b78ae9442c9/nova/volume/cinder.py#L105 | |
| 19:54:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: log vm and task state when vif plugging times out https://review.openstack.org/471459 | |
| 19:54:59 | sdague | efried: cool | |
| 19:55:25 | efried | sdague But makes a guy wonder why the client couldn't grab it from the auth, which is made from the context, which contains that request ID... | |
| 19:56:26 | sdague | it could be more magical | |
| 19:56:41 | sdague | but, being explicit here seems fine | |
| 19:57:14 | efried | ight | |
| 19:57:34 | sdague | given that it was a one line change for each service linkage, being explicit didn't seem so bad :) | |
| 19:58:23 | efried | sdague But a zero-line change would have been cooler ;-) | |
| 19:59:21 | mriedem | 0-line change means non-0 amount of debugging in 2 years time | |
| 19:59:24 | mriedem | b/c of magic | |
| 20:00:34 | sdague | mriedem: ++ | |
| 20:00:35 | efried | sdague How/where does global_id get set on a RequestContext? | |
| 20:00:54 | sdague | it's a property which is either global_request_id or request_id | |
| 20:01:07 | sdague | so it prefers global_request_id if that was sent in | |
| 20:01:17 | sdague | otherwise it uses the service request id | |
| 20:01:39 | sdague | request_id is generated as a uuid | |
| 20:01:49 | sdague | ok, TC / Pg thing, so won't be active here | |
| 20:03:41 | efried | sdague Thanks, I found it in the superclass (duh) | |
| 20:04:38 | openstackgerrit | Eric Fried proposed openstack/nova master: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | |
| 20:05:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Add service_token for nova-glance interaction https://review.openstack.org/419662 | |
| 20:11:42 | melwitt | mriedem: forgot to mention last week I wrote a functional test for that local delete allocations thing. looks like they do indeed get cleaned up upon init_host, but they would stay around if the compute host never returned https://review.openstack.org/#/c/470578 | |
| 20:23:12 | openstackgerrit | melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634 | |
| 20:23:13 | openstackgerrit | melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635 | |
| 20:28:55 | openstack | Launchpad bug 1696082 in OpenStack Compute (nova) "'nova.exception.NeutronAdminCredentialConfigurationInvalid" [Undecided,New] | |
| 20:28:55 | bauzas | mriedem: looks like https://bugs.launchpad.net/nova/+bug/1696082 is duplicate of https://bugs.launchpad.net/nova/+bug/1657774 ? | |
| 20:28:56 | openstack | Launchpad bug 1657774 in OpenStack Compute (nova) "Nova does not re-raise 401 Unauthorized received from Neutron for admin users" [Undecided,In progress] - Assigned to Roman Podoliaka (rpodolyaka) | |
| 20:30:33 | mriedem | shrug | |
| 20:30:35 | mriedem | TC meeting | |
| 20:30:44 | mriedem | melwitt: ok | |
| 20:30:58 | mriedem | melwitt: how's that bottom change on the quotas series going? | |
| 20:31:24 | artom | Speaking of bottom changes | |
| 20:31:36 | artom | mriedem, can I get you to put tagged attach on your review list again? | |
| 20:31:47 | bauzas | mriedem: dang. ok | |
| 20:32:13 | melwitt | mriedem: it's next on my list today, I just updated the placement stuff, working on the quotas thing now. when we talked at the summit, dansmith was saying it would be good to gate the "recheck quota" logic on a config option, in case someone doesn't mind quota going over during races | |
| 20:32:46 | melwitt | since a quota recheck is a re-count of resources, not a lightweight thing | |
| 20:33:26 | melwitt | so I was thinking of adding a config option also in the bottom change | |
| 20:34:02 | mriedem | artom: it already is, with 100 other things | |
| 20:34:15 | artom | mriedem, fair enough, thanks :) | |
| 20:35:07 | mriedem | bauzas: shrug == idk and don't have time to look, but maybe | |
| 20:35:32 | bauzas | mriedem: no worries, I see you active on the PG thingy | |
| 20:35:34 | artom | bauzas, don't think so - first is error 500, second one is error 400 | |
| 20:36:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Check if domain is persistent before detaching devices https://review.openstack.org/441204 | |
| 20:36:25 | artom | bauzas, but the message definitely is the same... | |
| 20:36:47 | artom | I feel like we should just close the first one with not enough info | |
| 20:36:53 | dansmith | melwitt: I think that's a good plan yeah | |
| 20:36:54 | bauzas | artom: yup, looks like both are related, just the fact that the API for nova list transforms it into 400 | |
| 20:37:00 | artom | You can't just paste a stack trace into launchpad and call it a bug report | |
| 20:37:17 | mriedem | maybe just review https://review.openstack.org/#/c/422696/ | |
| 20:38:06 | melwitt | cool, thanks dansmith | |
| 20:38:31 | mriedem | melwitt: i thought your bottom change was sabout instance groups quota? | |
| 20:38:37 | mriedem | wouldn't a config patch be separate? | |
| 20:38:53 | mriedem | oh wait, | |
| 20:39:01 | mriedem | the recheck quota thing in that patch, now i get it | |
| 20:39:11 | mriedem | and default that to what? | |
| 20:39:13 | mriedem | recheck by default? | |
| 20:39:22 | melwitt | yeah. I could do a separate patch for the config option but it would be by itself with nothing using it | |
| 20:39:32 | mriedem | just mix them together | |
| 20:39:33 | melwitt | yeah I was thinking recheck by default | |