| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-27 | |||
| 16:31:53 | melwitt | no, not yet. this is the second time they've found instances in that state but we don't yet know how it's happening | |
| 16:32:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Always retrieve network information if available https://review.openstack.org/564444 | |
| 16:32:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Retrieve tunneled status in '_get_physnet_info' https://review.openstack.org/564445 | |
| 16:32:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add NUMACell.network_metadata https://review.openstack.org/564439 | |
| 16:32:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add NUMATopologyLimits.networks https://review.openstack.org/575486 | |
| 16:32:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Start accounting for networks in NUMA placement https://review.openstack.org/564448 | |
| 16:32:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add RequestSpec.network_metadata https://review.openstack.org/564442 | |
| 16:32:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Start utilizing RequestSpec.network_metadata https://review.openstack.org/564452 | |
| 16:32:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Add '[neutron] physnets' and related options https://review.openstack.org/564440 | |
| 16:32:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start populating NUMACell.network_metadata field https://review.openstack.org/564441 | |
| 16:33:05 | mriedem | melwitt: and they don't have any errors in the logs? | |
| 16:33:53 | mriedem | dansmith: stephenfin: did you guys ever talk about the physnet/tunneled modeling stuff? | |
| 16:34:23 | stephenfin | mriedem: Nope. I didn't get a chance today. Can we do that tomorrow afternoon? | |
| 16:34:35 | mriedem | i can | |
| 16:35:04 | stephenfin | Cool. The above was simply reworking based on the comments I did have. I can respin based on whatever comes out of that tomorrow | |
| 16:35:19 | stephenfin | Though I did (hopefully) explain what I'm doing with additional comments where I set those fields | |
| 16:36:39 | dansmith | I'm out the rest of the week | |
| 16:36:56 | dansmith | but I think you guys can handle it | |
| 16:37:06 | stephenfin | (y) | |
| 16:37:40 | stephenfin | Cool. I'll give you a shout tomorrow morning (for you) so, mriedem | |
| 16:56:58 | openstackgerrit | Merged openstack/os-traits master: Add release note link in README https://review.openstack.org/578444 | |
| 17:06:26 | mriedem | i'm not sure i can handle it | |
| 17:07:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_member* methods https://review.openstack.org/578488 | |
| 17:19:49 | tssurya | dansmith: do we have a cells meeting today? | |
| 17:19:57 | dansmith | tssurya: I dunno, do we? | |
| 17:20:19 | tssurya | I don't have much (working on the spec implementation and POCs) | |
| 17:21:28 | melwitt | I don't have anything meeting worthy | |
| 17:21:31 | tssurya | ah and this funny but legit bug, : https://bugs.launchpad.net/nova/+bug/1778515 I guess it should be a doc fix | |
| 17:21:31 | openstack | Launchpad bug 1778515 in OpenStack Compute (nova) "nova-manage list_cells doesn't work if some special characters are in the passwords" [Undecided,New] - Assigned to Surya Seetharaman (tssurya) | |
| 17:21:32 | dansmith | okay I don't have anything | |
| 17:21:43 | tssurya | unless someone has better ideas | |
| 17:21:59 | melwitt | we talked about that a bit yesterday | |
| 17:22:12 | tssurya | oh just saw mriedem's comment | |
| 17:23:29 | melwitt | tssurya: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-06-26.log.html#t2018-06-26T21:23:37 | |
| 17:23:35 | mriedem | i haven't gone over the spec updates yet | |
| 17:23:38 | tssurya | okay then I don't have anything else :) thanks | |
| 17:23:55 | mriedem | i would appreciate reviews on this regression bug fix which is cells related https://review.openstack.org/#/c/576161/ | |
| 17:24:05 | mriedem | since that regression was backported and released in queens as well | |
| 17:24:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_delete method https://review.openstack.org/578223 | |
| 17:24:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_member* methods https://review.openstack.org/578488 | |
| 17:24:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove remaining legacy DB API instance_group* methods https://review.openstack.org/578497 | |
| 17:27:02 | mriedem | dansmith: this was the queens backport for tssurya's fix to ignore down cells while listing https://review.openstack.org/#/c/578152/ - you might want to hit that before heading out | |
| 17:27:19 | dansmith | ah cool thanks | |
| 17:27:19 | mriedem | tssurya: i assume you'll also want that in pike, although cern is at queens now so maybe you personally don't care | |
| 17:27:30 | mriedem | but others might | |
| 17:27:54 | mriedem | i think mgagne is going to queens as well | |
| 17:27:58 | mriedem | not sure about godaddy | |
| 17:28:01 | mriedem | or nectar | |
| 17:28:02 | tssurya | mriedem: true we only want things in queens now :D | |
| 17:28:14 | mriedem | SpamapS: are you guys going to queens for cells v2? | |
| 17:28:18 | mriedem | multi-cell i mean | |
| 17:28:37 | mgagne | mriedem: we are far from starting the process but that's the goal yes | |
| 17:28:46 | mriedem | ok | |
| 17:29:07 | dansmith | mriedem: ah yeah I had looked at that patch of yours and forgot to circle back after the typo | |
| 17:29:08 | tssurya | mriedem: I will put a backport out to pike as well , once the queens one gets merged just in case | |
| 17:29:09 | dansmith | mah bad | |
| 17:30:18 | mriedem | while i'm begging for stable reviews, this is a simple latent bug fix, hitting people in the ops list (that's how it came up) https://review.openstack.org/#/q/Iaaf7f68d6874fd5d6e737e7d2bc589ea4a048fee | |
| 17:30:44 | mriedem | we screw up reschedules for a failed resize if you're using the multitenant aggregates filter | |
| 17:30:53 | mriedem | tenant isolation filter i mean | |
| 17:37:04 | dansmith | I guess I should highlight this to the cells people too: https://review.openstack.org/#/c/578163/ | |
| 17:37:06 | dansmith | tssurya: ^ | |
| 18:31:35 | openstackgerrit | Merged openstack/os-vif master: Add release note link in README https://review.openstack.org/578446 | |
| 18:36:16 | mriedem | mordred: melwitt: dansmith: i've gone over the 'handling a down cell' spec again, comments inline but i think it's mostly good, just asking for more detail in a few spots - also wondering about handling queued_for_delete instances when listing/showing and when the cell comes back up for those | |
| 18:36:21 | mriedem | https://review.openstack.org/#/c/557369/ | |
| 18:38:49 | mriedem | tl;dr i think for queued_for_delete instances we want to list/show those as UNKNOWN, | |
| 18:39:09 | mriedem | and then once the cell is back up, the status on those goes back to whatever it's reported in the cell db, could be ACTIVE, SHUTDOWN, etc | |
| 18:39:27 | mriedem | oh which reminds me, when do we clear the queued_for_delete flag once it's set.... | |
| 18:42:10 | mriedem | uh, i think we might just not want to do queued_for_delete at all, | |
| 18:42:31 | mriedem | and 500 or 503 if you try to delete an instance in a down cell, like any other action on a server that's in a down cell like PUT /servers/{id} | |
| 18:49:04 | mriedem | random aside - can we delete this compat code from RT now? https://github.com/openstack/nova/blob/c8b93fa2493dce82ef4c0b1e7a503ba9b81c2e86/nova/compute/resource_tracker.py#L1206 | |
| 18:49:16 | mriedem | if you're in rocky, you shouldn't have ocata computes | |
| 18:49:57 | melwitt | mriedem: well, blocking a delete is new and undesirable behavior that we haven't had before. that's my first thought | |
| 18:50:13 | mriedem | it's not new if your database is down | |
| 18:50:16 | mriedem | which is the case here | |
| 18:51:14 | mriedem | and it's not like once the cell computes come back up they can up-call to the instance mappings table to see, oh while i was dead the user wanted to delete this instance | |
| 18:51:23 | melwitt | so the pre cells v2 behavior when nova-api is up but the database is down is what, a 500? | |
| 18:51:31 | mriedem | melwitt: i'm sure | |
| 18:52:39 | mriedem | queued_for_delete means, to me anyway, that we have some kind of periodic in the control plane services that is trying to hit the cells for instance mappings that are queued for delete and once it reaches a cell that's back up, it does the actual delete | |
| 18:52:42 | mriedem | that's not being proposed | |
| 18:53:36 | mriedem | sort of like how the _reclaim_queued_deletes periodic works in the nova-compute service for SOFT_DELETED instances | |
| 18:53:41 | melwitt | yeah, I get where you're coming from on that. and I'm trying to remember why then we wanted queued_for_delete in the first place. the ability to list/show "deleted" instance shells? since 'novs list --deleted' is a thing you can do | |
| 18:54:01 | melwitt | *nova list --deleted | |
| 18:54:11 | mriedem | maybe, but the spec calls it out as being needed for quota calculation | |
| 18:54:51 | melwitt | okay, I don't think it was supposed to, that is, we had thought we could piggyback the quota stuff on that column. the column itself had other purpose prior to that | |
| 18:55:05 | mriedem | well i guess it's not explicitly linking that in, but it says it's needed for nova list, nova show and nova boot | |
| 18:55:43 | mriedem | "``nova boot`` should not succeed if that project has any living VMs in the | |
| 18:55:43 | mriedem | down cell until an all-cell-iteration independent solution for quota | |
| 18:55:43 | mriedem | calculation is implemented through `quotas using placement`_." | |
| 18:55:53 | melwitt | yeah. my guess is that it's for the case where an instance was deleted in the past from some cell, then that cell goes down, then someone does 'nova list --deleted', we'd have no way of knowing deleted or not without access to the cell database | |
| 18:56:02 | mriedem | *living* VMs means, i assume, that queued_for_delete would be considered there during the quota calculation | |
| 18:57:25 | melwitt | because if an instance was deleted from a cell in the past, if the cell goes down, without queued_for_delete the instance would reappear in the nova list | |
| 18:57:33 | melwitt | if we're going off instance mappings and nothing else | |
| 18:58:52 | mriedem | so by that logic, we'd always need to set InstanceMapping.queued_for_delete=True when deleting an instance even if the cell is up and we actually do delete the instance from the cell | |
| 18:59:43 | melwitt | yeah, I thought that was how it's supposed to work. mark queued_for_delete=True when a delete request comes in, and that just means "the user intended to delete this" and does not indicate success or not | |
| 19:02:23 | mriedem | ok, and we never set it back to queued_for_delete=False? | |
| 19:02:28 | mriedem | that's what is bothering me | |
| 19:02:29 | melwitt | and that "queued" was a misnomer in that there's no queuing system at the API level for deletes. maybe better named "delete_requested" or something. guess we should double check with tssurya and dansmith. | |
| 19:02:35 | melwitt | I don't think so, no | |
| 19:03:13 | mriedem | we've said in the spec that we're just not going to include instances from down cells if there are filter parameters, | |
| 19:03:30 | mriedem | we could just consider 'nova list --deleted' as a filter parameter that we're not going to include results from down cells | |