| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 14:54:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 14:54:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 14:54:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 14:54:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 14:54:52 | bauzas | lyarwood: /me clicks anyway | |
| 14:54:52 | bauzas | lyarwood: /me clicks anyway | |
| 14:55:21 | lyarwood | ah crap the patch below hasn't actually landed | |
| 14:55:22 | lyarwood | ah crap the patch below hasn't actually landed | |
| 14:55:32 | lyarwood | sorry I thought it had, this isn't urgent in that case | |
| 14:55:32 | lyarwood | sorry I thought it had, this isn't urgent in that case | |
| 15:23:23 | melwitt | lyarwood: ack, I can look | |
| 15:23:23 | melwitt | lyarwood: ack, I can look | |
| 15:54:58 | stephenfin | Is it just me, or is nova-grenade-multinode failing 100% right now? | |
| 15:54:59 | stephenfin | Is it just me, or is nova-grenade-multinode failing 100% right now? | |
| 15:55:11 | stephenfin | * stephenfin compares two jobs | |
| 15:56:19 | stephenfin | Yeah, looks like Ubuntu is broken | |
| 15:56:19 | stephenfin | Yeah, looks like Ubuntu is broken | |
| 15:56:20 | stephenfin | Failed to start rtslib-fb-targetctl.service: Unit rtslib-fb-targetctl.service is not loaded properly: Exec format error. | |
| 15:56:21 | stephenfin | Failed to start rtslib-fb-targetctl.service: Unit rtslib-fb-targetctl.service is not loaded properly: Exec format error. | |
| 15:57:11 | lyarwood | yeah it is | |
| 15:57:11 | lyarwood | yeah it is | |
| 15:57:55 | lyarwood | stephenfin: https://bugs.launchpad.net/devstack/+bug/1926411 https://review.opendev.org/c/openstack/devstack/+/788429 | |
| 15:57:57 | openstack | Launchpad bug 1926411 in devstack "legacy bionic based dsvm jobs failing with Failed to start rtslib-fb-targetctl.service" [Undecided,In progress] | |
| 15:57:57 | lyarwood | stephenfin: https://bugs.launchpad.net/devstack/+bug/1926411 https://review.opendev.org/c/openstack/devstack/+/788429 | |
| 15:57:57 | openstack | Launchpad bug 1926411 in devstack "legacy bionic based dsvm jobs failing with Failed to start rtslib-fb-targetctl.service" [Undecided,In progress] | |
| 15:58:32 | stephenfin | aha, thanks | |
| 15:58:32 | stephenfin | aha, thanks | |
| 15:58:43 | stephenfin | I saw something about bionic scroll by earlier but didn't realize it was related | |
| 15:58:44 | stephenfin | I saw something about bionic scroll by earlier but didn't realize it was related | |
| 15:58:57 | lyarwood | argh and that's blocked by https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 15:59:03 | lyarwood | argh and that's blocked by https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 15:59:03 | openstack | Launchpad bug 1926434 in devstack "devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" [High,In progress] | |
| 15:59:08 | lyarwood | and https://review.opendev.org/c/openstack/devstack/+/620198 | |
| 15:59:09 | lyarwood | and https://review.opendev.org/c/openstack/devstack/+/620198 | |
| 16:12:16 | lyarwood | stephenfin: oh and https://review.opendev.org/c/openstack/nova/+/778885 hasn't landed yet but is also seeing failures related to https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 16:12:20 | openstack | Launchpad bug 1926434 in devstack "devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" [High,In progress] | |
| 16:12:20 | lyarwood | stephenfin: oh and https://review.opendev.org/c/openstack/nova/+/778885 hasn't landed yet but is also seeing failures related to https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 16:12:20 | openstack | Launchpad bug 1926434 in devstack "devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" [High,In progress] | |
| 16:56:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: QoS minimum guaranteed packet rate https://review.opendev.org/c/openstack/nova-specs/+/785014 | |
| 18:21:13 | openstackgerrit | Jessie Lass proposed openstack/nova master: Add emulation support if host arch != guest arch. https://review.opendev.org/c/openstack/nova/+/772156 | |
| 18:21:13 | openstackgerrit | Jessie Lass proposed openstack/nova master: Add emulation support if host arch != guest arch. https://review.opendev.org/c/openstack/nova/+/772156 | |
| 19:59:51 | clarkb | Hello nova. Hitting a weird error that doesn't happen 100% of the time. https://zuul.opendev.org/t/openstack/build/c7c698c95f2b4590ba95f14a6c97ffc0/log/nodepool/openstack/screen-n-api.txt#1735-1805 seems like when doing a server create somtimes nova forgets about the keystone domain? | |
| 19:59:51 | clarkb | Hello nova. Hitting a weird error that doesn't happen 100% of the time. https://zuul.opendev.org/t/openstack/build/c7c698c95f2b4590ba95f14a6c97ffc0/log/nodepool/openstack/screen-n-api.txt#1735-1805 seems like when doing a server create somtimes nova forgets about the keystone domain? | |
| 20:00:45 | clarkb | note this would've been run against master devstack (which is master nova). And the rough sequence of events that sometimes trips it is: `source /opt/devstack/openrc admin admin && openstack --os-project-name demo --os-username demo server create --flavor cirros256 --image $cirros_image unmanaged-vm --network public` | |
| 20:00:45 | clarkb | note this would've been run against master devstack (which is master nova). And the rough sequence of events that sometimes trips it is: `source /opt/devstack/openrc admin admin && openstack --os-project-name demo --os-username demo server create --flavor cirros256 --image $cirros_image unmanaged-vm --network public` | |
| 20:01:25 | clarkb | one suspicion for why it doesn't fail 100% of the time is that the traceback shows nova hitting _refresh_neutron_extensions_cache() which maybe isn't called on every request? | |
| 20:01:25 | clarkb | one suspicion for why it doesn't fail 100% of the time is that the traceback shows nova hitting _refresh_neutron_extensions_cache() which maybe isn't called on every request? | |
| 20:01:44 | clarkb | note that the devstack openrc should be providing the keystone user and project domain values | |
| 20:01:44 | clarkb | note that the devstack openrc should be providing the keystone user and project domain values | |
| 20:02:13 | clarkb | Is this familar to anyone? I'm happy to help debug further but quickly running out of thread to pull on | |
| 20:02:13 | clarkb | Is this familar to anyone? I'm happy to help debug further but quickly running out of thread to pull on | |
| 21:24:56 | openstackgerrit | Merged openstack/nova stable/train: libvirt: 'video.vram' property must be an integer https://review.opendev.org/c/openstack/nova/+/757618 | |
| 21:24:56 | openstackgerrit | Merged openstack/nova stable/train: libvirt: 'video.vram' property must be an integer https://review.opendev.org/c/openstack/nova/+/757618 | |
| 21:25:18 | gmann | clarkb: nova work on default domain, also there is no recent change on devstack side to remove default domain or so. | |
| 21:25:18 | gmann | clarkb: nova work on default domain, also there is no recent change on devstack side to remove default domain or so. | |
| 21:25:28 | gmann | i am not sure why that is happening. | |
| 21:25:28 | gmann | i am not sure why that is happening. | |
| 21:32:54 | andrewbogott | This is mostly a question for dansmith but I welcome advice from anyone :) I've just upgraded my install to V and all of my hypervisors are saying "Current Nova version does not support computes older than Ussuri but the minimum compute service level in your cell is 40..." | |
| 21:32:54 | andrewbogott | This is mostly a question for dansmith but I welcome advice from anyone :) I've just upgraded my install to V and all of my hypervisors are saying "Current Nova version does not support computes older than Ussuri but the minimum compute service level in your cell is 40..." | |
| 21:33:22 | andrewbogott | A totally reasonable warning except I cannot for the live of me figure out /why/ that's the minimum service level. All my nodes are updated, | |
| 21:33:22 | andrewbogott | A totally reasonable warning except I cannot for the live of me figure out /why/ that's the minimum service level. All my nodes are updated, | |
| 21:33:49 | andrewbogott | I've read the code, looked in all my DBs, no idea where that 40 is coming from. Can you tell me what the process is for calculating that? Is it cached someplace? | |
| 21:33:50 | andrewbogott | I've read the code, looked in all my DBs, no idea where that 40 is coming from. Can you tell me what the process is for calculating that? Is it cached someplace? | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:54 | andrewbogott | on startup | |
| 21:39:54 | andrewbogott | on startup | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |