| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-28 | |||
| 15:57:57 | openstack | Launchpad bug 1926411 in devstack "legacy bionic based dsvm jobs failing with Failed to start rtslib-fb-targetctl.service" [Undecided,In progress] | |
| 15:57:57 | lyarwood | stephenfin: https://bugs.launchpad.net/devstack/+bug/1926411 https://review.opendev.org/c/openstack/devstack/+/788429 | |
| 15:57:57 | openstack | Launchpad bug 1926411 in devstack "legacy bionic based dsvm jobs failing with Failed to start rtslib-fb-targetctl.service" [Undecided,In progress] | |
| 15:58:32 | stephenfin | aha, thanks | |
| 15:58:32 | stephenfin | aha, thanks | |
| 15:58:43 | stephenfin | I saw something about bionic scroll by earlier but didn't realize it was related | |
| 15:58:44 | stephenfin | I saw something about bionic scroll by earlier but didn't realize it was related | |
| 15:58:57 | lyarwood | argh and that's blocked by https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 15:59:03 | lyarwood | argh and that's blocked by https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 15:59:03 | openstack | Launchpad bug 1926434 in devstack "devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" [High,In progress] | |
| 15:59:08 | lyarwood | and https://review.opendev.org/c/openstack/devstack/+/620198 | |
| 15:59:09 | lyarwood | and https://review.opendev.org/c/openstack/devstack/+/620198 | |
| 16:12:16 | lyarwood | stephenfin: oh and https://review.opendev.org/c/openstack/nova/+/778885 hasn't landed yet but is also seeing failures related to https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 16:12:20 | openstack | Launchpad bug 1926434 in devstack "devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" [High,In progress] | |
| 16:12:20 | lyarwood | stephenfin: oh and https://review.opendev.org/c/openstack/nova/+/778885 hasn't landed yet but is also seeing failures related to https://bugs.launchpad.net/devstack/+bug/1926434 | |
| 16:12:20 | openstack | Launchpad bug 1926434 in devstack "devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" [High,In progress] | |
| 16:56:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: QoS minimum guaranteed packet rate https://review.opendev.org/c/openstack/nova-specs/+/785014 | |
| 18:21:13 | openstackgerrit | Jessie Lass proposed openstack/nova master: Add emulation support if host arch != guest arch. https://review.opendev.org/c/openstack/nova/+/772156 | |
| 18:21:13 | openstackgerrit | Jessie Lass proposed openstack/nova master: Add emulation support if host arch != guest arch. https://review.opendev.org/c/openstack/nova/+/772156 | |
| 19:59:51 | clarkb | Hello nova. Hitting a weird error that doesn't happen 100% of the time. https://zuul.opendev.org/t/openstack/build/c7c698c95f2b4590ba95f14a6c97ffc0/log/nodepool/openstack/screen-n-api.txt#1735-1805 seems like when doing a server create somtimes nova forgets about the keystone domain? | |
| 19:59:51 | clarkb | Hello nova. Hitting a weird error that doesn't happen 100% of the time. https://zuul.opendev.org/t/openstack/build/c7c698c95f2b4590ba95f14a6c97ffc0/log/nodepool/openstack/screen-n-api.txt#1735-1805 seems like when doing a server create somtimes nova forgets about the keystone domain? | |
| 20:00:45 | clarkb | note this would've been run against master devstack (which is master nova). And the rough sequence of events that sometimes trips it is: `source /opt/devstack/openrc admin admin && openstack --os-project-name demo --os-username demo server create --flavor cirros256 --image $cirros_image unmanaged-vm --network public` | |
| 20:00:45 | clarkb | note this would've been run against master devstack (which is master nova). And the rough sequence of events that sometimes trips it is: `source /opt/devstack/openrc admin admin && openstack --os-project-name demo --os-username demo server create --flavor cirros256 --image $cirros_image unmanaged-vm --network public` | |
| 20:01:25 | clarkb | one suspicion for why it doesn't fail 100% of the time is that the traceback shows nova hitting _refresh_neutron_extensions_cache() which maybe isn't called on every request? | |
| 20:01:25 | clarkb | one suspicion for why it doesn't fail 100% of the time is that the traceback shows nova hitting _refresh_neutron_extensions_cache() which maybe isn't called on every request? | |
| 20:01:44 | clarkb | note that the devstack openrc should be providing the keystone user and project domain values | |
| 20:01:44 | clarkb | note that the devstack openrc should be providing the keystone user and project domain values | |
| 20:02:13 | clarkb | Is this familar to anyone? I'm happy to help debug further but quickly running out of thread to pull on | |
| 20:02:13 | clarkb | Is this familar to anyone? I'm happy to help debug further but quickly running out of thread to pull on | |
| 21:24:56 | openstackgerrit | Merged openstack/nova stable/train: libvirt: 'video.vram' property must be an integer https://review.opendev.org/c/openstack/nova/+/757618 | |
| 21:24:56 | openstackgerrit | Merged openstack/nova stable/train: libvirt: 'video.vram' property must be an integer https://review.opendev.org/c/openstack/nova/+/757618 | |
| 21:25:18 | gmann | clarkb: nova work on default domain, also there is no recent change on devstack side to remove default domain or so. | |
| 21:25:18 | gmann | clarkb: nova work on default domain, also there is no recent change on devstack side to remove default domain or so. | |
| 21:25:28 | gmann | i am not sure why that is happening. | |
| 21:25:28 | gmann | i am not sure why that is happening. | |
| 21:32:54 | andrewbogott | This is mostly a question for dansmith but I welcome advice from anyone :) I've just upgraded my install to V and all of my hypervisors are saying "Current Nova version does not support computes older than Ussuri but the minimum compute service level in your cell is 40..." | |
| 21:32:54 | andrewbogott | This is mostly a question for dansmith but I welcome advice from anyone :) I've just upgraded my install to V and all of my hypervisors are saying "Current Nova version does not support computes older than Ussuri but the minimum compute service level in your cell is 40..." | |
| 21:33:22 | andrewbogott | A totally reasonable warning except I cannot for the live of me figure out /why/ that's the minimum service level. All my nodes are updated, | |
| 21:33:22 | andrewbogott | A totally reasonable warning except I cannot for the live of me figure out /why/ that's the minimum service level. All my nodes are updated, | |
| 21:33:49 | andrewbogott | I've read the code, looked in all my DBs, no idea where that 40 is coming from. Can you tell me what the process is for calculating that? Is it cached someplace? | |
| 21:33:50 | andrewbogott | I've read the code, looked in all my DBs, no idea where that 40 is coming from. Can you tell me what the process is for calculating that? Is it cached someplace? | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:03 | dansmith | it should be stored on the service record | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:34:23 | dansmith | and yeah I'd guess maybe some old record is being considered (or you have one node not updated like you think) | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:15 | andrewbogott | tell me more about 'the service record'? | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:35:34 | dansmith | the records in the services table in the db | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:16 | andrewbogott | so you would think that 'delete from services where topic='compute' and version != 53;' would fix it right? | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:25 | andrewbogott | (which I already did and it didn't) | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:38 | dansmith | no, I'd say look at them first and let's figure out what's going on | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:36:50 | dansmith | they're all the new version? | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:01 | andrewbogott | there were several entries in there from nodes that don't exist anymore referring to old versions | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:37:05 | andrewbogott | that's why I ran that delete. | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:07 | dansmith | okay, and you've restarted services? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:38:23 | andrewbogott | maybe not, which services would I need to restart? apis? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:37 | dansmith | where are you seeing the warning? | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:51 | andrewbogott | nova-compute logs | |
| 21:39:54 | andrewbogott | on startup | |
| 21:39:54 | andrewbogott | on startup | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:10 | andrewbogott | (so yeah, I've definitely restarted those) | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:40:25 | andrewbogott | well -- to be clear, I haven't restarted all my compute services, only the one I'm watching the logs for | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:05 | andrewbogott | but I definitely restarted them all after the version upgrade | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:41:39 | andrewbogott | I'm happy to hack in a debug line if there's someplace I can dump the name of the service it's finding that's sub-40 | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:42:09 | andrewbogott | (thank you, btw, for immediately responding to my unfunded tech support request!) | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:36 | dansmith | okay, well, tbh I barely remember how this works, and I don't think I wrote it | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:43:53 | dansmith | but restarting nova-compute should be plenty for nova-compute to re-survey things | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:01 | dansmith | do you have multiple cell dbs? | |
| 21:44:21 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:44:22 | andrewbogott | I have an empty cell0 and one actual in-use cell | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:15 | andrewbogott | I can also just ignore the warning since it's clearly harmless in the current version. Patch comments imply that it might be a hard error in the future though. | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:47 | dansmith | yeah, so all of the services run that code, not just nova-compute | |
| 21:45:59 | dansmith | right, hard error now | |
| 21:46:00 | dansmith | right, hard error now | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:10 | andrewbogott | ok, should probably sort it out before I hit that wall then | |
| 21:46:14 | andrewbogott | * andrewbogott frantically restarts things | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |
| 21:46:47 | dansmith | victoria is 52 according to the code | |