Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-27
15:25:55 fungi perfect, thanks! i'll move forward on drafting the advisory and get some input on the wording once i have it fleshed out
15:26:08 dansmith okay
15:26:53 openstackgerrit qingszhao proposed openstack/os-vif master: Add release note link in README https://review.openstack.org/578446
15:52:30 openstackgerrit Matt Riedemann proposed openstack/os-vif master: Add release note link in README https://review.openstack.org/578446
15:52:47 mriedem dansmith: ok
15:52:57 mriedem dansmith: also this was ok https://review.openstack.org/#/c/578192/
15:53:01 mriedem the failing functional test was a known thing
15:57:43 dansmith mriedem: I saw
15:57:52 dansmith you can imagine why I assumed... :)
16:03:44 mriedem replies on some of https://review.openstack.org/#/c/563375/17/nova/objects/instance_group.py
16:03:48 mriedem i think we're mostly on the same page though
16:19:41 dansmith mriedem: replied
16:19:55 dansmith I plied, then I plied again
16:26:31 melwitt okay, just got off the phone with the rdo cloud peeps (kforde) and the state of the instance was that it had no instance mapping (no row) and it only existed as a build request. there was no other reference to the instance, and it was stuck in vm_state 'building' and task_state 'scheduling'
16:27:08 melwitt it couldn't be deleted because it failed to find an instance mapping (no instance with uuid found)
16:27:48 mriedem dansmith: ok i'm not sure i'm following all of the lazy-load stuff you're talking about
16:27:57 mriedem not sure if you mean the joinedload on the db object?
16:28:07 mriedem when you say lazy-load, i think obj_load_attr()
16:28:26 mriedem melwitt: ok was that instance created in like mitaka?
16:29:23 melwitt mriedem: I don't think so, no. the deployment is ocata and he said there's been no upgrade or change around the time the instance was created (it was created a month ago)
16:30:18 melwitt do we currently have code that will handle this case, no instance mapping but build request only, for a delete? (mostly rhetorical, will check the code)
16:30:34 mriedem we create the instance mapping at the same time that we create the build request, in the api
16:30:52 mriedem so it doesn't really make sense, unless we have something that accidentally deleted the instance mapping, but we don't do that anywhere in ocata as far as i know
16:30:54 melwitt hrm.
16:31:15 mriedem and is it reproducable?
16:31:37 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1010
16:31:53 melwitt no, not yet. this is the second time they've found instances in that state but we don't yet know how it's happening
16:32:45 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Always retrieve network information if available https://review.openstack.org/564444
16:32:46 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Retrieve tunneled status in '_get_physnet_info' https://review.openstack.org/564445
16:32:47 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMACell.network_metadata https://review.openstack.org/564439
16:32:49 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Start accounting for networks in NUMA placement https://review.openstack.org/564448
16:32:49 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMATopologyLimits.networks https://review.openstack.org/575486
16:32:50 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add RequestSpec.network_metadata https://review.openstack.org/564442
16:32:51 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Start utilizing RequestSpec.network_metadata https://review.openstack.org/564452
16:32:53 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Add '[neutron] physnets' and related options https://review.openstack.org/564440
16:32:54 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start populating NUMACell.network_metadata field https://review.openstack.org/564441
16:33:05 mriedem melwitt: and they don't have any errors in the logs?
16:33:53 mriedem dansmith: stephenfin: did you guys ever talk about the physnet/tunneled modeling stuff?
16:34:23 stephenfin mriedem: Nope. I didn't get a chance today. Can we do that tomorrow afternoon?
16:34:35 mriedem i can
16:35:04 stephenfin Cool. The above was simply reworking based on the comments I did have. I can respin based on whatever comes out of that tomorrow
16:35:19 stephenfin Though I did (hopefully) explain what I'm doing with additional comments where I set those fields
16:36:39 dansmith I'm out the rest of the week
16:36:56 dansmith but I think you guys can handle it
16:37:06 stephenfin (y)
16:37:40 stephenfin Cool. I'll give you a shout tomorrow morning (for you) so, mriedem
16:56:58 openstackgerrit Merged openstack/os-traits master: Add release note link in README https://review.openstack.org/578444
17:06:26 mriedem i'm not sure i can handle it
17:07:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_member* methods https://review.openstack.org/578488
17:19:49 tssurya dansmith: do we have a cells meeting today?
17:19:57 dansmith tssurya: I dunno, do we?
17:20:19 tssurya I don't have much (working on the spec implementation and POCs)
17:21:28 melwitt I don't have anything meeting worthy
17:21:31 openstack Launchpad bug 1778515 in OpenStack Compute (nova) "nova-manage list_cells doesn't work if some special characters are in the passwords" [Undecided,New] - Assigned to Surya Seetharaman (tssurya)
17:21:31 tssurya ah and this funny but legit bug, : https://bugs.launchpad.net/nova/+bug/1778515 I guess it should be a doc fix
17:21:32 dansmith okay I don't have anything
17:21:43 tssurya unless someone has better ideas
17:21:59 melwitt we talked about that a bit yesterday
17:22:12 tssurya oh just saw mriedem's comment
17:23:29 melwitt tssurya: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-06-26.log.html#t2018-06-26T21:23:37
17:23:35 mriedem i haven't gone over the spec updates yet
17:23:38 tssurya okay then I don't have anything else :) thanks
17:23:55 mriedem i would appreciate reviews on this regression bug fix which is cells related https://review.openstack.org/#/c/576161/
17:24:05 mriedem since that regression was backported and released in queens as well
17:24:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_delete method https://review.openstack.org/578223
17:24:47 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_member* methods https://review.openstack.org/578488
17:24:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove remaining legacy DB API instance_group* methods https://review.openstack.org/578497
17:27:02 mriedem dansmith: this was the queens backport for tssurya's fix to ignore down cells while listing https://review.openstack.org/#/c/578152/ - you might want to hit that before heading out
17:27:19 mriedem tssurya: i assume you'll also want that in pike, although cern is at queens now so maybe you personally don't care
17:27:19 dansmith ah cool thanks
17:27:30 mriedem but others might
17:27:54 mriedem i think mgagne is going to queens as well
17:27:58 mriedem not sure about godaddy
17:28:01 mriedem or nectar
17:28:02 tssurya mriedem: true we only want things in queens now :D
17:28:14 mriedem SpamapS: are you guys going to queens for cells v2?
17:28:18 mriedem multi-cell i mean
17:28:37 mgagne mriedem: we are far from starting the process but that's the goal yes
17:28:46 mriedem ok
17:29:07 dansmith mriedem: ah yeah I had looked at that patch of yours and forgot to circle back after the typo
17:29:08 tssurya mriedem: I will put a backport out to pike as well , once the queens one gets merged just in case
17:29:09 dansmith mah bad
17:30:18 mriedem while i'm begging for stable reviews, this is a simple latent bug fix, hitting people in the ops list (that's how it came up) https://review.openstack.org/#/q/Iaaf7f68d6874fd5d6e737e7d2bc589ea4a048fee
17:30:44 mriedem we screw up reschedules for a failed resize if you're using the multitenant aggregates filter
17:30:53 mriedem tenant isolation filter i mean
17:37:04 dansmith I guess I should highlight this to the cells people too: https://review.openstack.org/#/c/578163/
17:37:06 dansmith tssurya: ^
18:31:35 openstackgerrit Merged openstack/os-vif master: Add release note link in README https://review.openstack.org/578446
18:36:16 mriedem mordred: melwitt: dansmith: i've gone over the 'handling a down cell' spec again, comments inline but i think it's mostly good, just asking for more detail in a few spots - also wondering about handling queued_for_delete instances when listing/showing and when the cell comes back up for those
18:36:21 mriedem https://review.openstack.org/#/c/557369/
18:38:49 mriedem tl;dr i think for queued_for_delete instances we want to list/show those as UNKNOWN,
18:39:09 mriedem and then once the cell is back up, the status on those goes back to whatever it's reported in the cell db, could be ACTIVE, SHUTDOWN, etc
18:39:27 mriedem oh which reminds me, when do we clear the queued_for_delete flag once it's set....
18:42:10 mriedem uh, i think we might just not want to do queued_for_delete at all,
18:42:31 mriedem and 500 or 503 if you try to delete an instance in a down cell, like any other action on a server that's in a down cell like PUT /servers/{id}
18:49:04 mriedem random aside - can we delete this compat code from RT now? https://github.com/openstack/nova/blob/c8b93fa2493dce82ef4c0b1e7a503ba9b81c2e86/nova/compute/resource_tracker.py#L1206
18:49:16 mriedem if you're in rocky, you shouldn't have ocata computes
18:49:57 melwitt mriedem: well, blocking a delete is new and undesirable behavior that we haven't had before. that's my first thought
18:50:13 mriedem it's not new if your database is down
18:50:16 mriedem which is the case here

Earlier   Later