| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-20 | |||
| 10:19:09 | johnthetubaguy | so if you do controllers upgrade, then wait for them all to be the same | |
| 10:19:12 | johnthetubaguy | its quite close | |
| 10:19:17 | johnthetubaguy | but I get your point | |
| 10:19:39 | mgoddard | service_version = service_obj.get_minimum_version_all_cells( | |
| 10:19:42 | mgoddard | context.get_admin_context(), ['nova-compute']) | |
| 10:19:58 | mgoddard | I'd quite like a command that does that, compares with the latest version and says yes/no | |
| 10:20:13 | johnthetubaguy | yeah, that would be a good nova-manage cmd | |
| 10:20:15 | mgoddard | or even waits with a timeout | |
| 10:20:43 | mgoddard | I'm leaning towards short sleep + nova RFE at this point :) | |
| 10:21:29 | johnthetubaguy | RFE? what is that | |
| 10:22:30 | mgoddard | blueprint | |
| 10:22:37 | mgoddard | :p | |
| 10:22:49 | johnthetubaguy | yeah, I am wondering about nova status as a place for this kind of thing | |
| 10:22:56 | johnthetubaguy | "is every upgraded" check, makes sense | |
| 10:23:06 | mgoddard | yeah | |
| 10:24:57 | johnthetubaguy | i am not seeing anything useful just yet | |
| 10:27:25 | johnthetubaguy | I am sure we used to have a nova-manage cmd for this, I guess we killed it | |
| 10:27:49 | mgoddard | that's a shame | |
| 10:30:25 | johnthetubaguy | https://github.com/openstack/nova/blob/fc8cd8f3b346c8f53e2dfc8e3de9fdcaedb0d35d/nova/cmd/manage.py#L613 | |
| 10:30:38 | johnthetubaguy | although doesn't seem like that had the version either | |
| 10:32:52 | mgoddard | that was just pushed into the API wasn't it? | |
| 10:34:00 | mgoddard | if the version was available in the service API that would be fine, but ideally I'd want a way to discover what the version should be | |
| 10:34:35 | mgoddard | Looks like it takes 10+ seconds to update the service version in an AIO setup | |
| 10:36:11 | johnthetubaguy | there is a reporting delay config | |
| 10:36:33 | johnthetubaguy | yeah, so I can't find anything useful in the API or nova-manage, need to add a new thing | |
| 10:36:59 | johnthetubaguy | I mean SIGHUP should be enough, but not sure how we do that via docker? | |
| 10:37:39 | mgoddard | no SIGHUP, it's busted. Need to restart | |
| 10:37:59 | mgoddard | we do that already in kolla, the problem is we're not waiting long enough to actually uncap | |
| 10:38:19 | johnthetubaguy | ah, right | |
| 10:39:11 | johnthetubaguy | https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/servicegroup/api.py#L35 | |
| 10:39:13 | mgoddard | so I think my band aid is 'sleep ${nova_compute_startup_delay:-30}' | |
| 10:39:25 | johnthetubaguy | so we do wait 5 seconds before reporting | |
| 10:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/609460 | |
| 10:39:51 | johnthetubaguy | that is once its called, and in a timer that might trigger at somepoint after that | |
| 10:40:26 | mgoddard | although I think the thing I'm waiting for happens here: https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/service.py#L77 | |
| 10:41:20 | johnthetubaguy | init_host is quite expensive | |
| 10:41:33 | johnthetubaguy | it might be just the cost of waiting for that to finish | |
| 10:41:50 | johnthetubaguy | but yeah, its not affected by what I was pointing at | |
| 10:43:33 | mgoddard | would you call this a bug or blueprint? | |
| 10:44:27 | johnthetubaguy | its either really, the release notes are independent these days, so it matters little | |
| 10:44:32 | johnthetubaguy | (to me) | |
| 10:45:19 | johnthetubaguy | the API change would be a spec, but that is a separate issue, a quick fix to nova-manage to list all the services not currently at the latest version would be a simple enough patch | |
| 10:46:20 | johnthetubaguy | you have to target a cell with hosts in it, so maybe its a cell command? | |
| 10:46:47 | johnthetubaguy | get_non_upgrade_cell_computes() or ideally some less rubbish name | |
| 10:48:26 | mgoddard | so there are two cases. API, scheduler & super conductor need to wait for all computes. Cell conductor & computes need to wait just for the cell | |
| 10:48:51 | mgoddard | similar logic to https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/compute/rpcapi.py#L412 | |
| 10:49:41 | johnthetubaguy | hmm... eek | |
| 10:49:52 | mgoddard | which does mean that we shouldn't restart the API etc until all cells have been upgraded | |
| 10:50:03 | mgoddard | quite the ballet we have going on here | |
| 10:51:01 | johnthetubaguy | yeah, I think that is correct... | |
| 10:51:20 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix test_flavors to run with correct microversion https://review.opendev.org/666536 | |
| 10:51:39 | johnthetubaguy | I think its that --cascade argument, one just checks the local cell, the other checks all cells | |
| 10:52:12 | johnthetubaguy | it is correct though, no compute <-> compute communication between cells | |
| 10:52:32 | johnthetubaguy | and after the pin is lifted, compute still accepts older version of requests coming from the API | |
| 10:52:57 | johnthetubaguy | till everyone is done... then boom off we go | |
| 10:53:26 | johnthetubaguy | does kolla give the child cells a connection to the API database? | |
| 10:53:36 | johnthetubaguy | that is going to breaking this logic, interestingly | |
| 10:53:54 | mgoddard | it did until recently | |
| 10:54:08 | mgoddard | actually that was something different | |
| 10:54:27 | johnthetubaguy | yeah, there is a strong assumption about that now | |
| 10:54:30 | mgoddard | we don't have a conductor/superconductor split currently | |
| 10:55:04 | mgoddard | not even sure how that distinction is made TBH, is it just the presence of an API DB connection? | |
| 10:55:36 | johnthetubaguy | yeah | |
| 10:55:40 | johnthetubaguy | in the nova.conf | |
| 10:56:12 | johnthetubaguy | if not empty, check all cells is what it does there | |
| 10:56:48 | johnthetubaguy | I mean nova-compute shouldn't have any database strings, I think, so this is about the conductor really | |
| 10:57:12 | mgoddard | right | |
| 10:58:20 | johnthetubaguy | hmm, is that a recent change in kolla-ansible, removing the db conf from nova-compute? | |
| 11:03:49 | mgoddard | johnthetubaguy: yeah, but backported | |
| 11:08:41 | johnthetubaguy | mgoddard: ah, cool, I guess I might see that fix soon | |
| 11:12:21 | jroll | kashyap: thanks! | |
| 11:23:59 | kashyap | jroll: So, from discussion in #qemu (OFTC network): | |
| 11:24:31 | kashyap | jroll: It _is_ on disk (as you guessed): libvirt stores the state under: /var/lib/libvirt/swtpm/. It is *not* encrypted yet; however, libvirt is *capable* of it | |
| 11:25:01 | kashyap | jroll: And the relevant libvirt/QEMU devs seem to discuss the design of it, as we speak | |
| 11:26:52 | kashyap | jroll: LOL, and the QEMU dev asks me to file a libvirt RFE. Now I need to read up the relevant context to file a meaningful bug | |
| 11:29:04 | jroll | kashyap: oof, sorry I got you roped into that :/ | |
| 11:33:33 | kashyap | jroll: Apparently _another_ RFE is also required, for "swtpm" -- to let us pass keys without them being on disk. | |
| 11:34:30 | kashyap | jroll: Regardless, your question was interesting, and I learned something new. I'll let you know once the RFE bugs are filed (see how carefully I dodge the question of _who_ might file them :D) | |
| 11:35:27 | jroll | kashyap: thanks for your help with it, I do appreciate it :) | |
| 11:36:04 | jroll | now the question is, do we implement tpm passthrough in nova for the time being? or do I carry that patch downstream? :/ | |
| 11:36:59 | kashyap | jroll: Might want to also ask on the upstream list -- some operators might chime in | |
| 11:37:14 | jroll | kashyap: sorry, yes, I meant I'll email the upstream list today :) | |
| 11:38:18 | kashyap | Ah, okay :-) | |
| 11:39:32 | efried | stephenfin: I sleep. I'm actually (usually) very careful about getting enough sleep, as I find it makes all other things better. | |
| 11:51:33 | kashyap | Yes! | |
| 11:51:39 | aspiers | stephenfin: you need to try stestr run -n path.to.module.Class.test, it really does work ... | |
| 11:52:12 | kashyap | efried: In your copious free time, I vehemently suggest to read this (by a real sleep scientist, engagingly written): https://www.penguin.co.uk/books/295/295665/why-we-sleep/9780141983769.html | |
| 11:52:43 | aspiers | /dev/cft: device full | |
| 11:52:58 | kashyap | [Don't let "impressions" from the title come in your way. He writes, and speaks very engagingly and is _full_ of "signal".] | |
| 11:53:21 | efried | kashyap: oo | |
| 11:56:04 | mgoddard | that is an excellent book | |
| 11:59:18 | kashyap | Yeah, as the author, Matt Walker, dedicated 20 years (and still on going) studying sleep. | |
| 12:01:49 | stephenfin | aspiers: o rly? | |
| 12:03:33 | efried | aspiers, stephenfin: Are we highlighting -n (don't do discovery) here? | |
| 12:03:39 | stephenfin | Yuuup | |
| 12:03:44 | efried | ++ | |
| 12:03:44 | stephenfin | I knew it worked for files | |
| 12:03:50 | aspiers | see also https://github.com/mtreinish/stestr/issues/219 | |
| 12:03:51 | stephenfin | but not for Python module paths | |
| 12:04:28 | efried | Yeah, I usually can't be bothered to type the thing out, so I run it once with a substring and no -n, and then copy the test name from that output for subsequent runs with -n | |