| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-22 | |||
| 18:48:45 | artom | Random thought - someone should probably remove efried and mriedem from https://review.opendev.org/#/admin/groups/25,members, no? | |
| 18:48:53 | artom | Nothing against you fellas, obviously :) | |
| 19:00:32 | mriedem | i'm removed myseflf | |
| 19:00:38 | mriedem | *myself even | |
| 19:01:14 | mriedem | and https://review.opendev.org/#/admin/groups/1936,members | |
| 19:07:58 | artom | mriedem, RIP in pieces :( | |
| 19:08:21 | artom | I was randomly linking that page to someone who needed core review | |
| 19:08:34 | artom | This isn't a coup, is what I'm saying :) | |
| 19:21:03 | mriedem | yup, np | |
| 20:22:57 | jralbert | in 'nova hypervisor-stats', and have no entries in the allocations table in the nova_api database. Running nova-manage placement heal_allocations for affected instances returns an error that "Compute host <compute_node> could not be found", but all the compute nodes are visible in nova-manage cell_v2 list_hosts. Has anyone encountered this? Any | |
| 20:22:57 | jralbert | I've raised this question in #openstack and so far received no response; anybody here have any advice? I've just upgraded from Rocky to Stein, and Placement seems to have gotten very confused about resources in-use vs available. A number of compute nodes with running instances which are known to 'openstack server show' are reporting 0 running_vms | |
| 20:22:58 | jralbert | suggestions on how to troubleshoot it further? | |
| 20:39:15 | melwitt | jralbert: it sounds like your nova.instances records instance.host and instance.node aren't matching any records in nova.compute_nodes 'host' and 'hypervisor_hostname', as for why you're getting Compute host could not be found | |
| 20:40:30 | melwitt | did you replace any nova-compute services during the upgrade with different hostnames or something like that? | |
| 20:41:36 | jralbert | melwitt: Thanks, I'll have a look at those tables. We didn't change any hostnames during the upgrade, although because it was being orchestrated by OSA there may have been steps taken we weren't fully aware of | |
| 20:43:49 | melwitt | ok. I can't tell you how things got into that state but just for the sake of understanding why heal_allocations won't work: it will look up the compute node for an instance by querying for the instance.host and instance.node in the nova.compute_nodes table and if it finds no match, you'll get the Compute host could not be found error | |
| 20:45:32 | melwitt | I'm not aware of any change from rocky to stein that could cause something like this. are you running with integrated placement (in nova) or did you split out to running extracted placement I wonder? | |
| 20:46:38 | jralbert | I'm not 100% sure what integrated/extracted placement means, but I can find out - we're relying mostly on OSA defaults with some overrides, but that's not one I'm aware of. | |
| 20:46:44 | melwitt | if you've moved to extracted placement, there are additional steps you need to follow for an upgrade, linked from here: https://docs.openstack.org/releasenotes/nova/stein.html#prelude | |
| 20:47:02 | jralbert | Interestingly, there is a bit of a hint in the tables: http://paste.openstack.org/show/796222/ | |
| 20:47:05 | melwitt | https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html | |
| 20:47:29 | jralbert | The compute nodes get '.openstack.local' hostnames in instances, but a genuine FQDN in compute_nodes | |
| 20:48:33 | melwitt | the reason I ask about extracted placement is because you said your nova_api.allocations table is empty. did you already move to a separate placement db? do you have placement.allocations records? | |
| 20:49:13 | jralbert | Sorry, the allocations table isn't completely empty - it's just missing *some* instances on *some* compute nodes | |
| 20:49:30 | melwitt | ok, I think that difference in the nodename is causing the failure to lookup the compute nodes for the heal_allocations command | |
| 20:49:46 | jralbert | we became aware of this because our instance/vcpu metrics suddenly dropped by about half during the upgrade, but all the instances were still running that had been | |
| 20:50:19 | melwitt | this is where I see the lookup by host and node for heal_allocations https://github.com/openstack/nova/blob/stable/stein/nova/cmd/manage.py#L1829 | |
| 20:50:42 | jralbert | So would I safely be able to update those definitions directly in the database, or is there a better-bounded API/CLI approach? | |
| 20:52:32 | jralbert | I guess I would have hoped that compute nodes would be identified by uuids since names (especially hostnames) often produce collisions like this, but I'm guessing there's a legacy reason why this is the way it is. | |
| 20:55:09 | melwitt | yeah, they are not linked by uuid. I don't know the history behind it. probably something like since you can delete/create services via the API, if we didn't link by name, you'd lose the association for all your instances even if the hostname is the same (service create makes a new uuid) | |
| 20:56:21 | jralbert | Yep, I can imagine. We too are victims of our own legacy here: our compute nodes always had shortnames rather than FQDNs, and I'm pretty sure that's how we got burned here | |
| 20:56:27 | melwitt | I think you'd have to update manually in the database for this (please backup first). I'm looking to see where the hypervisor_hostname comes from when the compute_nodes record is initially created, to see how it happened and whether it will happen again if you were to delete/create the nova-compute service later on | |
| 20:57:53 | jralbert | I actually think I can maybe do this without futzing with the database, come to think of it: if I let nova-compute discover the node's FQDN and register itself as a "new" compute-service, it'll probably also create a new compute node record with FQDN, and ideally trigger a new resource allocation update. I'll try this with one node to see how it | |
| 20:57:54 | jralbert | goes - otherwise, I'll update in the database | |
| 20:58:05 | jralbert | Thank you very much for taking the time to help me with this today, I really appreciate it | |
| 20:58:28 | melwitt | yeah, that would be the ideal way. fingers crossed that it'll match what your instances already have stored in their host/node | |
| 20:58:44 | jralbert | To be discovered! | |
| 22:11:51 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: testing tempest copy-image test https://review.opendev.org/742547 | |
| 22:44:31 | efried | artom: It's not out of the question that I'll still review something here or there. But yeah, soon. | |
| 22:45:59 | artom | efried, still addicted, eh? I thought you quit cold-turkey ;) | |
| 22:46:15 | artom | But ack, you're an adult, you can do the needful when it's time | |
| 22:46:18 | artom | ;) | |
| 22:46:18 | efried | I've quit almost nothing in my life cold turkey | |
| 22:47:26 | efried | With the possible exception of running. | |
| 22:47:45 | efried | I actually had an orthopedic surgeon tell me I was too white to run marathons. | |
| 22:47:56 | artom | Hah, you told me that once | |
| 22:48:10 | artom | I remember laughing like an idiot at the most recent Denver PTG | |
| 22:49:23 | efried | orite, because you were limping, s'how that conversation started. All recovered by now I assume (like three years later) | |
| 22:50:12 | artom | Yeah, back to running :) | |
| 22:50:21 | artom | No full marathons, but I've done a couple of halfs | |
| 22:51:09 | efried | nice. I had a half where all the stars aligned, averaged 7:30/mi, reverse split. My best run ever. | |
| 22:51:22 | efried | (which tells you all you need to know about my running talent) | |
| #openstack-nova - 2020-07-23 | |||
| 00:05:01 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of networks policies https://review.opendev.org/742558 | |
| 00:11:28 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in networks policy https://review.opendev.org/742561 | |
| 00:31:48 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566 | |
| 00:32:45 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of networks policies https://review.opendev.org/742558 | |
| 00:32:53 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in networks policy https://review.opendev.org/742561 | |
| 00:35:38 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566 | |
| 00:36:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566 | |
| 00:46:24 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568 | |
| 00:54:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in FIP policy https://review.opendev.org/742570 | |
| 00:55:16 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568 | |
| 01:50:13 | openstackgerrit | Merged openstack/nova master: libvirt: Remove two-level loop https://review.opendev.org/741920 | |
| 02:07:30 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 05:28:24 | openstackgerrit | Merged openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | |
| 05:28:31 | openstackgerrit | Merged openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997 | |
| 05:52:51 | openstackgerrit | Merged openstack/nova master: objects: Update keypairs when saving an instance https://review.opendev.org/683043 | |
| 06:36:41 | openstackgerrit | Marek Lyčka proposed openstack/nova-specs master: Enable REST quiescing https://review.opendev.org/702810 | |
| 06:57:48 | openstackgerrit | Brin Zhang proposed openstack/nova master: Delete ARQs by UUID if Cyborg ARQ bind fails. https://review.opendev.org/716186 | |
| 07:13:53 | gibi | good morning | |
| 07:15:58 | brinzhang_ | gibi: good moning^ | |
| 07:16:42 | brinzhang_ | gibi: I was update the https://review.opendev.org/#/c/716186/16/nova/exception.py patch, but as your comment, after I update, taht will be an error, you can see in http://paste.openstack.org/show/796234/ | |
| 07:17:10 | brinzhang_ | do you know why? the code looks good | |
| 07:22:29 | gibi | brinzhang_: will check in 20 minutes (on a call right now) | |
| 07:23:13 | brinzhang_ | gibi: thanks, I missed a parameter in __init__, will update | |
| 07:30:56 | openstackgerrit | Brin Zhang proposed openstack/nova master: Delete ARQs by UUID if Cyborg ARQ bind fails. https://review.opendev.org/716186 | |
| 07:30:57 | brinzhang_ | gibi: you can check all the update in the latest patch now. | |
| 08:10:20 | kashyap | lyarwood: I see the VIR_ERR_DEVICE_MISSING is merged; sorry, yetesrday I had the draft ACK sitting but didn't hit 'submit'. I see it's taken care by three cores :) | |
| 08:29:14 | gibi | brinzhang_: thanks, I'm +2 | |
| 08:29:34 | brinzhang_ | gibi: coolthanks | |
| 08:33:56 | gibi | stephenfin: I put https://review.opendev.org/#/q/topic:bp/add-emulated-virtual-tpm to the empty runway slot for the next two weeks | |
| 08:34:08 | gibi | stephenfin: is that OK for you? | |
| 08:35:10 | gibi | stephenfin: I will be on PTO next week but I hope others will check | |
| 08:35:27 | gibi | and I can pick that up the week after | |
| 08:45:58 | gibi | dansmith: Do I see it correclty that the work in https://blueprints.launchpad.net/nova/+spec/rbd-glance-multistore has been completed? | |
| 09:18:15 | aarents | Hi nova, | |
| 09:18:44 | aarents | gibi, when it will be possible, can you advise on last comments of https://review.opendev.org/#/c/696084/3 as my change conflicts with yours here: https://review.opendev.org/#/c/739246/1 as lyarwood mentioned | |
| 09:22:58 | gibi | aarents: looking... | |
| 09:34:56 | gibi | aarents: go ahead with your patch my patch can be the last in the chain | |
| 09:38:06 | aarents | gibi: ok thanks | |
| 09:39:51 | stephenfin | gibi: Sure. Could I ask you to grab the base patch in the series again before you go? You reviewed it before and it's just a rebase | |
| 09:46:15 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: objects: Update keypairs when saving an instance https://review.opendev.org/742631 | |
| 09:47:51 | jkulik | Do you see any chance of implementing some kind of sharding on the nova-compute level? With the vmwareapi driver managing a cluster of hypervisors, there are hundreds of VMs behind every nova-compute node and at times this leads to tasks queuing up a lot. | |
| 09:48:53 | bauzas | gibi: stephenfin: fwiw, I'll enter the Phantom Zone starting end of next week for the next 3 weeks | |
| 09:49:18 | bauzas | gibi: so when you are back, I'll be off (IIUC your vacation time) | |
| 09:49:19 | jkulik | It would be nice, if we could spawn multiple nova-compute processes per cluster and I was thinking of letting them work on VMs based on the UUID so they don't interfere so easily. | |
| 09:50:05 | openstackgerrit | Merged openstack/nova master: compute: Do not allow rescue attempts using volume snapshot images https://review.opendev.org/729261 | |
| 09:50:23 | stephenfin | bauzas: What's the chances of having you work the vTPM stuff with lyarwood? I can handle the 'provider.yaml' stuff (and anything else) since I didn't write it, but not vTPM | |
| 09:50:47 | bauzas | stephenfin: by 'work', you mean 'reviews' ? | |
| 09:50:55 | stephenfin | Correct | |