| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-20 | |||
| 01:11:08 | openstackgerrit | sean mooney proposed openstack/os-traits master: add GOP, NONE and VIRTIO grapics models https://review.opendev.org/665830 | |
| 02:02:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove deprecated arguments in db sync command https://review.opendev.org/666470 | |
| 06:11:43 | openstackgerrit | mashuting proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/521690 | |
| 08:31:35 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Grab fresh power state info from the driver https://review.opendev.org/665975 | |
| 08:40:17 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Grab fresh power state info from the driver https://review.opendev.org/665975 | |
| 08:41:24 | stephenfin | efried: When do you _sleep_? | |
| 08:41:47 | stephenfin | :) | |
| 08:59:53 | moshele | sean-k-mooney | |
| 09:00:02 | moshele | sean-k-mooney: hi | |
| 09:02:13 | ohwhyosa | Morning guys and gals! | |
| 09:15:20 | kashyap | jroll: Just saw the ping in scrollback; I don't know the answer to where the keys are stored. I'll inquire and let you know. | |
| 09:22:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove global state from the FakeDriver https://review.opendev.org/656709 | |
| 09:23:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enhance service restart in functional env https://review.opendev.org/512552 | |
| 09:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add functional test coverage for bug 1724172 https://review.opendev.org/512553 | |
| 09:23:28 | openstack | bug 1724172 in OpenStack Compute (nova) rocky "Allocation of an evacuated instance is not cleaned on the source host if instance is not defined on the hypervisor" [Medium,Confirmed] https://launchpad.net/bugs/1724172 | |
| 09:38:14 | openstackgerrit | guangchao.pan proposed openstack/nova master: Add a name for the automatically created volume https://review.opendev.org/660096 | |
| 10:16:26 | mgoddard | Hello novatons, I have a question about upgrades using [upgrade_levels]compute=auto. Is there a programatic way to know when all computes are running the new RPC version, so that I can restart services to remove the version cap? | |
| 10:16:33 | mgoddard | johnthetubaguy: ^ | |
| 10:17:14 | mgoddard | I'm looking for something like 'nova-manage wait for computes [--all-cells]' | |
| 10:17:18 | johnthetubaguy | there is the services API, does that not return you the version? | |
| 10:17:22 | mgoddard | nope | |
| 10:17:46 | mgoddard | the only potentially useful thing there is updated_at, but that's a bit kludgey | |
| 10:17:48 | johnthetubaguy | checking if its in a new microversion | |
| 10:17:56 | mgoddard | ah | |
| 10:18:15 | mgoddard | I'd quite like to not need to care what the latest version is | |
| 10:19:09 | johnthetubaguy | so if you do controllers upgrade, then wait for them all to be the same | |
| 10:19:12 | johnthetubaguy | its quite close | |
| 10:19:17 | johnthetubaguy | but I get your point | |
| 10:19:39 | mgoddard | service_version = service_obj.get_minimum_version_all_cells( | |
| 10:19:42 | mgoddard | context.get_admin_context(), ['nova-compute']) | |
| 10:19:58 | mgoddard | I'd quite like a command that does that, compares with the latest version and says yes/no | |
| 10:20:13 | johnthetubaguy | yeah, that would be a good nova-manage cmd | |
| 10:20:15 | mgoddard | or even waits with a timeout | |
| 10:20:43 | mgoddard | I'm leaning towards short sleep + nova RFE at this point :) | |
| 10:21:29 | johnthetubaguy | RFE? what is that | |
| 10:22:30 | mgoddard | blueprint | |
| 10:22:37 | mgoddard | :p | |
| 10:22:49 | johnthetubaguy | yeah, I am wondering about nova status as a place for this kind of thing | |
| 10:22:56 | johnthetubaguy | "is every upgraded" check, makes sense | |
| 10:23:06 | mgoddard | yeah | |
| 10:24:57 | johnthetubaguy | i am not seeing anything useful just yet | |
| 10:27:25 | johnthetubaguy | I am sure we used to have a nova-manage cmd for this, I guess we killed it | |
| 10:27:49 | mgoddard | that's a shame | |
| 10:30:25 | johnthetubaguy | https://github.com/openstack/nova/blob/fc8cd8f3b346c8f53e2dfc8e3de9fdcaedb0d35d/nova/cmd/manage.py#L613 | |
| 10:30:38 | johnthetubaguy | although doesn't seem like that had the version either | |
| 10:32:52 | mgoddard | that was just pushed into the API wasn't it? | |
| 10:34:00 | mgoddard | if the version was available in the service API that would be fine, but ideally I'd want a way to discover what the version should be | |
| 10:34:35 | mgoddard | Looks like it takes 10+ seconds to update the service version in an AIO setup | |
| 10:36:11 | johnthetubaguy | there is a reporting delay config | |
| 10:36:33 | johnthetubaguy | yeah, so I can't find anything useful in the API or nova-manage, need to add a new thing | |
| 10:36:59 | johnthetubaguy | I mean SIGHUP should be enough, but not sure how we do that via docker? | |
| 10:37:39 | mgoddard | no SIGHUP, it's busted. Need to restart | |
| 10:37:59 | mgoddard | we do that already in kolla, the problem is we're not waiting long enough to actually uncap | |
| 10:38:19 | johnthetubaguy | ah, right | |
| 10:39:11 | johnthetubaguy | https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/servicegroup/api.py#L35 | |
| 10:39:13 | mgoddard | so I think my band aid is 'sleep ${nova_compute_startup_delay:-30}' | |
| 10:39:25 | johnthetubaguy | so we do wait 5 seconds before reporting | |
| 10:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/609460 | |
| 10:39:51 | johnthetubaguy | that is once its called, and in a timer that might trigger at somepoint after that | |
| 10:40:26 | mgoddard | although I think the thing I'm waiting for happens here: https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/service.py#L77 | |
| 10:41:20 | johnthetubaguy | init_host is quite expensive | |
| 10:41:33 | johnthetubaguy | it might be just the cost of waiting for that to finish | |
| 10:41:50 | johnthetubaguy | but yeah, its not affected by what I was pointing at | |
| 10:43:33 | mgoddard | would you call this a bug or blueprint? | |
| 10:44:27 | johnthetubaguy | its either really, the release notes are independent these days, so it matters little | |
| 10:44:32 | johnthetubaguy | (to me) | |
| 10:45:19 | johnthetubaguy | the API change would be a spec, but that is a separate issue, a quick fix to nova-manage to list all the services not currently at the latest version would be a simple enough patch | |
| 10:46:20 | johnthetubaguy | you have to target a cell with hosts in it, so maybe its a cell command? | |
| 10:46:47 | johnthetubaguy | get_non_upgrade_cell_computes() or ideally some less rubbish name | |
| 10:48:26 | mgoddard | so there are two cases. API, scheduler & super conductor need to wait for all computes. Cell conductor & computes need to wait just for the cell | |
| 10:48:51 | mgoddard | similar logic to https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/compute/rpcapi.py#L412 | |
| 10:49:41 | johnthetubaguy | hmm... eek | |
| 10:49:52 | mgoddard | which does mean that we shouldn't restart the API etc until all cells have been upgraded | |
| 10:50:03 | mgoddard | quite the ballet we have going on here | |
| 10:51:01 | johnthetubaguy | yeah, I think that is correct... | |
| 10:51:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix test_flavors to run with correct microversion https://review.opendev.org/666536 | |
| 10:51:39 | johnthetubaguy | I think its that --cascade argument, one just checks the local cell, the other checks all cells | |
| 10:52:12 | johnthetubaguy | it is correct though, no compute <-> compute communication between cells | |
| 10:52:32 | johnthetubaguy | and after the pin is lifted, compute still accepts older version of requests coming from the API | |
| 10:52:57 | johnthetubaguy | till everyone is done... then boom off we go | |
| 10:53:26 | johnthetubaguy | does kolla give the child cells a connection to the API database? | |
| 10:53:36 | johnthetubaguy | that is going to breaking this logic, interestingly | |
| 10:53:54 | mgoddard | it did until recently | |
| 10:54:08 | mgoddard | actually that was something different | |
| 10:54:27 | johnthetubaguy | yeah, there is a strong assumption about that now | |
| 10:54:30 | mgoddard | we don't have a conductor/superconductor split currently | |
| 10:55:04 | mgoddard | not even sure how that distinction is made TBH, is it just the presence of an API DB connection? | |
| 10:55:36 | johnthetubaguy | yeah | |
| 10:55:40 | johnthetubaguy | in the nova.conf | |
| 10:56:12 | johnthetubaguy | if not empty, check all cells is what it does there | |
| 10:56:48 | johnthetubaguy | I mean nova-compute shouldn't have any database strings, I think, so this is about the conductor really | |
| 10:57:12 | mgoddard | right | |
| 10:58:20 | johnthetubaguy | hmm, is that a recent change in kolla-ansible, removing the db conf from nova-compute? | |
| 11:03:49 | mgoddard | johnthetubaguy: yeah, but backported | |
| 11:08:41 | johnthetubaguy | mgoddard: ah, cool, I guess I might see that fix soon | |
| 11:12:21 | jroll | kashyap: thanks! | |
| 11:23:59 | kashyap | jroll: So, from discussion in #qemu (OFTC network): | |
| 11:24:31 | kashyap | jroll: It _is_ on disk (as you guessed): libvirt stores the state under: /var/lib/libvirt/swtpm/. It is *not* encrypted yet; however, libvirt is *capable* of it | |
| 11:25:01 | kashyap | jroll: And the relevant libvirt/QEMU devs seem to discuss the design of it, as we speak | |
| 11:26:52 | kashyap | jroll: LOL, and the QEMU dev asks me to file a libvirt RFE. Now I need to read up the relevant context to file a meaningful bug | |