| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-06 | |||
| 19:08:08 | sdague | stuff was probably behind it and failing | |
| 19:09:44 | mriedem | here is the oslo.log bug https://bugs.launchpad.net/oslo.log/+bug/1696213 | |
| 19:09:45 | openstack | Launchpad bug 1696213 in oslo.log "error summary formatter is putting ": NotImplementedError" in non-error log messages" [Undecided,New] | |
| 19:10:03 | mriedem | back to this rebuild failure | |
| 19:14:44 | mriedem | sdague: i see the problem | |
| 19:14:44 | mriedem | http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz#_Jun_06_13_19_17_300332 | |
| 19:14:49 | mriedem | vif plugging callback timeout | |
| 19:14:50 | mriedem | during rebuild | |
| 19:14:53 | mriedem | so the instance never goes active | |
| 19:15:26 | mriedem | which someone from heat reported a rebuild bug about vif plug timeouts recently | |
| 19:16:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1694371 | |
| 19:16:05 | openstack | Launchpad bug 1694371 in OpenStack Compute (nova) "test_stack_snapshot_restore failing intermittently" [Undecided,New] | |
| 19:17:54 | mriedem | kevinbenton: we probably need some help from the neutron side here https://bugs.launchpad.net/nova/+bug/1694371 | |
| 19:17:54 | openstack | Launchpad bug 1694371 in OpenStack Compute (nova) "Timeout while waiting for network-vif-plugged event during server rebuild" [Undecided,New] | |
| 19:18:06 | mriedem | kevinbenton: seeing rebuild tests timeout waiting for vif plugged events from neutron | |
| 19:22:19 | mriedem | dansmith: does it make sense to do anything in nova.objects.Instance.mutated_migration_context if there is no self.migration_context? | |
| 19:22:31 | mriedem | because we're lazy-loading 3 fields in there just to do nothing | |
| 19:23:03 | dansmith | um | |
| 19:23:11 | mriedem | i'm seeing this during a rebuild | |
| 19:23:13 | mriedem | where we're not migrating | |
| 19:23:21 | mriedem | but the rebuild code also handles evacuate, which is the migration | |
| 19:23:42 | mriedem | but it loos like we end up lazy-loading numa_topology, pci_requests and pci_devices one after another just to ultimately do nothing | |
| 19:24:48 | dansmith | yeah I guess not.. you see that debug log warning that there's no context too I guess? | |
| 19:25:27 | mriedem | http://logs.openstack.org/82/471082/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/796acb7/logs/screen-n-cpu.txt.gz#_Jun_06_13_14_02_547424 | |
| 19:25:31 | mriedem | yup | |
| 19:25:32 | cfriesen_ | mriedem: if you're not actually migrating I think it'd be safe to bypass the mutated_migration_context | |
| 19:25:35 | mriedem | see the next 4 there | |
| 19:25:45 | mriedem | it lazy-loads 3 fields, and then says "you're dumb" | |
| 19:26:19 | mriedem | and i believe each lazy-load on those fields is going to be a full pull from the db via conductor... | |
| 19:27:58 | kevinbenton | mriedem: in a rebuild what does nova tell neutron? | |
| 19:28:06 | kevinbenton | mriedem: because the host_id doesn't change, right? | |
| 19:28:48 | cfriesen_ | mriedem: can a rebuild end up picking a new host? | |
| 19:29:46 | cfriesen_ | (when we're not doing an evacuate, I mean) | |
| 19:30:23 | mriedem | cfriesen_: kevinbenton: no, same host | |
| 19:30:35 | cfriesen_ | mriedem:, right, we specify the host in that case | |
| 19:30:39 | mriedem | kevinbenton: so my theory is neutron is not sending the event maybe because the host doesn't change | |
| 19:30:57 | mriedem | but i'd think that would be 100% fail | |
| 19:31:28 | mriedem | so i wonder if we're hitting a race where neutron is queueing notifications and thinks it already told nova something about a given port | |
| 19:32:29 | kevinbenton | mriedem: we did have a recent change on the neutron side that reduced some port status changes when they weren't needed | |
| 19:32:58 | kevinbenton | mriedem: so it's possible that there is a race that was being covered by side effects from that before | |
| 19:33:35 | kevinbenton | mriedem: do you have a set of logs from one of these failures? | |
| 19:35:25 | mriedem | kevinbenton: yeah that bug linked above has a log dive from me | |
| 19:35:30 | mriedem | showing where things happen on the nova side | |
| 19:37:43 | efried | sdague Can you please help me understand whether/how global_request_id affordance is needed in https://review.openstack.org/#/c/412634/28/nova/image/glance.py ? | |
| 19:38:25 | efried | I'm kinda hoping it's implicitly part of the ksa session... | |
| 19:38:45 | efried | ...or auth | |
| 19:40:38 | efried | ...because of the context | |
| 19:49:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | |
| 19:49:50 | openstackgerrit | Eric Fried proposed openstack/nova master: Add service_token for nova-glance interaction https://review.openstack.org/419662 | |
| 19:51:26 | sdague | efried: it's not implicit | |
| 19:52:08 | sdague | efried: it's a single parameter to pass | |
| 19:52:08 | efried | sdague So I need to add the header explicitly? | |
| 19:52:17 | sdague | one sec | |
| 19:52:53 | sdague | efried: https://review.openstack.org/#/c/467242/6/nova/image/glance.py | |
| 19:53:10 | efried | sdague Right, I'm rebasing against that. | |
| 19:53:18 | efried | sdague But in the new version, we don't do any of that stuff. | |
| 19:53:32 | sdague | efried: yeh, so just add global_request_id=context.global_id | |
| 19:53:50 | sdague | to the constructor args for glanceclient.Client | |
| 19:54:23 | efried | sdague Okay, will do. Thanks. | |
| 19:54:43 | sdague | like this - https://github.com/openstack/nova/blob/2f712b84ecf8d149833dea17d2fa4b78ae9442c9/nova/volume/cinder.py#L105 | |
| 19:54:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: log vm and task state when vif plugging times out https://review.openstack.org/471459 | |
| 19:54:59 | sdague | efried: cool | |
| 19:55:25 | efried | sdague But makes a guy wonder why the client couldn't grab it from the auth, which is made from the context, which contains that request ID... | |
| 19:56:26 | sdague | it could be more magical | |
| 19:56:41 | sdague | but, being explicit here seems fine | |
| 19:57:14 | efried | ight | |
| 19:57:34 | sdague | given that it was a one line change for each service linkage, being explicit didn't seem so bad :) | |
| 19:58:23 | efried | sdague But a zero-line change would have been cooler ;-) | |
| 19:59:21 | mriedem | 0-line change means non-0 amount of debugging in 2 years time | |
| 19:59:24 | mriedem | b/c of magic | |
| 20:00:34 | sdague | mriedem: ++ | |
| 20:00:35 | efried | sdague How/where does global_id get set on a RequestContext? | |
| 20:00:54 | sdague | it's a property which is either global_request_id or request_id | |
| 20:01:07 | sdague | so it prefers global_request_id if that was sent in | |
| 20:01:17 | sdague | otherwise it uses the service request id | |
| 20:01:39 | sdague | request_id is generated as a uuid | |
| 20:01:49 | sdague | ok, TC / Pg thing, so won't be active here | |
| 20:03:41 | efried | sdague Thanks, I found it in the superclass (duh) | |
| 20:04:38 | openstackgerrit | Eric Fried proposed openstack/nova master: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | |
| 20:05:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Add service_token for nova-glance interaction https://review.openstack.org/419662 | |
| 20:11:42 | melwitt | mriedem: forgot to mention last week I wrote a functional test for that local delete allocations thing. looks like they do indeed get cleaned up upon init_host, but they would stay around if the compute host never returned https://review.openstack.org/#/c/470578 | |
| 20:23:12 | openstackgerrit | melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634 | |
| 20:23:13 | openstackgerrit | melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635 | |
| 20:28:55 | bauzas | mriedem: looks like https://bugs.launchpad.net/nova/+bug/1696082 is duplicate of https://bugs.launchpad.net/nova/+bug/1657774 ? | |
| 20:28:55 | openstack | Launchpad bug 1696082 in OpenStack Compute (nova) "'nova.exception.NeutronAdminCredentialConfigurationInvalid" [Undecided,New] | |
| 20:28:56 | openstack | Launchpad bug 1657774 in OpenStack Compute (nova) "Nova does not re-raise 401 Unauthorized received from Neutron for admin users" [Undecided,In progress] - Assigned to Roman Podoliaka (rpodolyaka) | |
| 20:30:33 | mriedem | shrug | |
| 20:30:35 | mriedem | TC meeting | |
| 20:30:44 | mriedem | melwitt: ok | |
| 20:30:58 | mriedem | melwitt: how's that bottom change on the quotas series going? | |
| 20:31:24 | artom | Speaking of bottom changes | |
| 20:31:36 | artom | mriedem, can I get you to put tagged attach on your review list again? | |
| 20:31:47 | bauzas | mriedem: dang. ok | |
| 20:32:13 | melwitt | mriedem: it's next on my list today, I just updated the placement stuff, working on the quotas thing now. when we talked at the summit, dansmith was saying it would be good to gate the "recheck quota" logic on a config option, in case someone doesn't mind quota going over during races | |
| 20:32:46 | melwitt | since a quota recheck is a re-count of resources, not a lightweight thing | |
| 20:33:26 | melwitt | so I was thinking of adding a config option also in the bottom change | |
| 20:34:02 | mriedem | artom: it already is, with 100 other things | |
| 20:34:15 | artom | mriedem, fair enough, thanks :) | |
| 20:35:07 | mriedem | bauzas: shrug == idk and don't have time to look, but maybe | |
| 20:35:32 | bauzas | mriedem: no worries, I see you active on the PG thingy | |