| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-17 | |||
| 23:19:04 | rm_work | err, /etc/nova/nova.conf | |
| 23:19:17 | rm_work | it seems to not be respecting all of the sql connection config | |
| 23:19:47 | rm_work | we use x509 auth for our sql connection, which is in the connection string in nova.conf as &ssl_cert=...&ssl_key=... | |
| 23:20:08 | rm_work | which works for the actual service, but the nova-manage db stuff doesn't seem to use it correctly | |
| 23:23:30 | rm_work | yeah, verified even with the config explicitly set, it's not respecting the x509 args, so the db migration scripts fail on the sql connection | |
| 23:28:20 | rm_work | hmmm, well that was my assumption, but maybe it's actually that it is missing some other sql permission? trying to dig in deeper, I just know I get this error when it tries to run `_get_marker_for_migrate_instances` | |
| 23:28:56 | rm_work | `(pymysql.err.OperationalError) (1045, "Access denied for user 'nova'@'sqlhost' (using password: YES)")` | |
| 23:29:29 | rm_work | but it does seem to be able to get SOME data, as it printed some numbers about the migration counts correctly | |
| #openstack-nova - 2020-11-18 | |||
| 00:10:52 | rm_work | ahh figured it out | |
| 00:11:15 | rm_work | the nova_api DB stores cell connection strings that need to be updated if the connection string changes at all | |
| 00:11:29 | rm_work | this seems a little awkward? but I assume it is necessary for some reason to cache this data here | |
| 00:13:50 | artom | rm_work, if you're familiar with why and how cells v2 came about, it makes sense | |
| 00:15:13 | rm_work | yeah so if we cycle the DB password, or the certs move, we need to make sure to update the cells first via `nova-manage cells_v2 update_cell` | |
| 00:19:07 | artom | rm_work, yep | |
| 03:12:44 | openstackgerrit | Brin Zhang proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/729563 | |
| 04:33:40 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241 | |
| 07:20:41 | openstackgerrit | Xinran WANG proposed openstack/nova-specs master: SRIOV Enabled Nic Support Specification https://review.opendev.org/742785 | |
| 08:04:33 | bauzas | good morning Nova | |
| 08:13:34 | tacco | Morning * | |
| 08:19:38 | gibi | o/ | |
| 08:32:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Add upgrade check about old computes https://review.opendev.org/761924 | |
| 08:32:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Improve error handling during service level check https://review.opendev.org/762471 | |
| 08:33:53 | eandersson | Is there a tool to clean up placement entries that no longer exists in nova? :D | |
| 08:34:28 | eandersson | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L1326 | |
| 08:35:04 | eandersson | It's not something super widespread, but I have 22 computes that have at least one of these ghost allocations. | |
| 08:36:14 | gibi | eandersson: $ nova-manage placement audit # https://docs.openstack.org/nova/latest/cli/nova-manage.html | |
| 08:37:21 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Fix released] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 08:37:21 | gibi | eandersson: https://bugs.launchpad.net/nova/+bug/1793569 | |
| 08:37:39 | eandersson | Nice. Guessing that there isn't a back-port on that one. ;D | |
| 08:38:28 | gibi | eandersson: there was backport proposals https://review.opendev.org/#/q/status:abandoned+topic:placement-audit-backport | |
| 08:38:47 | eandersson | Thanks yea just found it | |
| 08:40:15 | eandersson | I'll try it out in the lab | |
| 08:56:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Restore retrying the RPC connection to conductor https://review.opendev.org/763138 | |
| 09:09:47 | eandersson | Worked great in the lab | |
| 09:11:58 | gibi | eandersson: good to hear that | |
| 09:42:03 | lyarwood | Morning | |
| 10:29:09 | openstack | bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 | |
| 10:29:09 | openstackgerrit | Merged openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100 | |
| 10:32:57 | openstackgerrit | Merged openstack/nova master: Set instance host and drop migration under lock https://review.opendev.org/754815 | |
| 10:33:06 | openstackgerrit | Merged openstack/nova master: [Trivial]Add missing white space in the log message https://review.opendev.org/762804 | |
| 10:33:16 | openstackgerrit | Merged openstack/nova master: [docs] Fix a placement client's command https://review.opendev.org/762663 | |
| 10:35:53 | openstack | bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 | |
| 10:35:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Reproduce bug 1896463 in func env https://review.opendev.org/763152 | |
| 10:38:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Set instance host and drop migration under lock https://review.opendev.org/763154 | |
| 11:36:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Revert "Temporarily disable parts of heal port allocation test" https://review.opendev.org/755995 | |
| 11:41:14 | stephenfin | Does unshelving a server always result in that server going to ACTIVE or ERROR state? | |
| 11:42:20 | stephenfin | I'm adding a '--wait' option to 'openstack server unshelve' and was checking for 'SHUTOFF' status, but a quick check suggests we don't restore to the original state after unshelving https://review.opendev.org/#/c/763166/1/openstackclient/compute/v2/server.py@3713 | |
| 11:43:15 | stephenfin | So I guess I just need to check for 'ACTIVE'? | |
| 11:49:17 | sean-k-mooney | i shoudl go to active but if there is an error i dont knwo if it will go back to shelved or error | |
| 11:49:52 | sean-k-mooney | we wont restore the orgininal state as far as i know | |
| 11:56:57 | stephenfin | okay, I'll just check for ACTIVE so | |
| 12:24:30 | openstackgerrit | Shi Yan proposed openstack/nova master: Fix the instance_uuid for test https://review.opendev.org/763171 | |
| 13:58:49 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP libvirt: Allow the default machine type to be changed https://review.opendev.org/762199 | |
| 14:02:13 | aarents | Hi nova, | |
| 14:03:24 | aarents | lyarwood: can you confirm, when you can, that those both change are ok ? https://review.opendev.org/#/c/743708/ https://review.opendev.org/#/c/752463/2 thanks ! | |
| 15:10:49 | gmann | stephenfin: sean-k-mooney I agree on full list of hypervisors to project admin if no limit - https://review.opendev.org/#/c/763043/1/specs/wallaby/approved/modernize-os-hypervisors-api.rst@162 | |
| 15:11:29 | gmann | stephenfin: if you can update the other comments I am good on this. | |
| 15:18:37 | owalsh | artom, rm_work: FYI there is a better option for DB creds etc... for cell uris: https://docs.openstack.org/nova/latest/user/cells.html#template-urls-in-cell-mappings | |
| 15:21:32 | stephenfin | gmann: Cool, I will respin that shortly | |
| 15:21:49 | stephenfin | dansmith: Is 'nova refresh-network' worth porting to OSC? | |
| 15:22:03 | dansmith | stephenfin: isn't that a xen-specific thing? | |
| 15:22:16 | stephenfin | It uses the os-server-external-events extension | |
| 15:22:21 | stephenfin | "extension" | |
| 15:22:31 | stephenfin | is that XenAPI specific? | |
| 15:22:50 | stephenfin | surely not; that's what neutron uses iirc | |
| 15:23:19 | stephenfin | asking you specifically because you authored the novaclient change waaaay back https://review.opendev.org/#/c/74763/ | |
| 15:23:30 | dansmith | oh, then I'm thinking of something else | |
| 15:24:37 | owalsh | dansmith: re cell nova-conductor still requiring api db for affinity checks, should I assume that will eventually change? | |
| 15:24:39 | dansmith | oh, right, this probably really shouldn't even be a CLI thing, we probably added it there so we could poke it for testing | |
| 15:24:40 | rm_work | owalsh: oooo will take a look, thanks | |
| 15:24:59 | stephenfin | okay, that was my gut feel too; I'll skip it so | |
| 15:25:04 | stephenfin | dansmith++ thanks | |
| 15:26:19 | dansmith | stephenfin: I was thinking of this: https://docs.openstack.org/api-ref/compute/#reset-networking-on-a-server-resetnetwork-action | |
| 15:26:40 | dansmith | note the "no longer supported by any virt driver" because xen | |
| 15:26:56 | stephenfin | aha, yes, I've done work on that recently | |
| 15:28:47 | dansmith | owalsh: probably not any time soon.. it was hinging on affinity in placement, and I gather that people aren't interested in furthering those goals these days, | |
| 15:28:55 | dansmith | owalsh: so probably no change on the horizon | |
| 15:39:40 | owalsh | dansmith: ack, thanks, so it wouldn't be crazy to use these creds to create the cell mapping when deploying a new cell conductor? | |
| 15:40:09 | dansmith | owalsh: nope, that's reasonable at the moment I'd say | |
| 15:46:12 | owalsh | dansmith: ack, great. Also does metadata api need the api db config even when local_metadata_per_cell=True? | |
| 15:47:06 | dansmith | owalsh: shouldn't, no | |
| 15:48:08 | dansmith | owalsh: yeah, if that's set, we never run the mapping lookup, which is the api hit | |
| 15:48:25 | dansmith | owalsh: so unless there's a bug or something buried deep, that's the intent.. no api db hit if =True | |
| 15:49:21 | owalsh | dansmith: ack, yea, didn't initially make sense that local_metadata_per_cell existed when everything else just checks if the api db is configured, but need to support cell conductor & metadata api sharing nova.conf | |
| 15:53:27 | dansmith | owalsh: well, I think metadata was the first thing to need that determination, so we added a conf knob for it specifically.. it's also potentially security-related because the workloads can talk to it, so explicit-ness is good | |
| 15:53:47 | dansmith | owalsh: but we could deprecate that conf knob and actually just honor the presence of the api creds if you think it'd be better for consistency | |
| 15:58:11 | owalsh | dansmith: but that wouldn't work in cases where the cell conductor shares nova conf with cell metadata-api | |
| 15:58:32 | dansmith | cell conductor, yeah | |
| 16:01:25 | owalsh | dansmith: which goes back to the recommendation on the ML to follow the devstack approach to conf - it doesn't seem to work when using cell local metadata api... | |
| 16:02:01 | owalsh | since metadata-api always uses nova.conf, never nova-cellN.conf | |
| 16:02:36 | dansmith | owalsh: but devstack runs a global metadata api right? | |
| 16:03:02 | owalsh | dansmith: yup, doesn't seem to support cell local metadata api | |
| 16:03:31 | dansmith | devstack's goal, of course, is not to support every possible deployment arrangement... | |
| 16:03:59 | owalsh | dansmith: indeed, and for CI/dev running multiple cells makes sense but in production it does not... | |
| 16:04:09 | owalsh | multiple cells on the same host that is | |
| 16:04:45 | dansmith | I've lost track of what we're arguing about | |
| 16:06:14 | owalsh | dansmith: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018732.html | |
| 16:06:59 | owalsh | I'm saying we can't use a similar strategy to separate files | |
| 16:10:04 | dansmith | owalsh: okay I'm sorry, I've really lost track of who "we" and what the problem is. You're saying "we=tripleo" deploy a metadata per cell and can't, for some reason, point it at the cellN conductor's config file? | |
| 16:11:44 | owalsh | dansmith: rpms/debs in particular are the issue, but yes, can't point metadata-api at the conductor's config file | |