| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-27 | |||
| 10:03:09 | opendevreview | Balazs Gibizer proposed openstack/nova master: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810909 | |
| 10:43:27 | lyarwood | gibi: https://zuul.opendev.org/t/openstack/build/b607db4018bb4753839d6fd1202322b4/log/controller/logs/screen-q-svc.txt#29241 - nova-grenade-multinode appears to be failing 100% of the time due to a missing extension in Neutron, any ideas if anything has changed recently? | |
| 10:43:38 | gibi | lyarwood: looking | |
| 10:52:16 | gibi | lyarwood: this feels relevant https://review.opendev.org/c/openstack/neutron/+/809550 | |
| 10:52:44 | gibi | the revert landed ~1 day ago | |
| 10:53:04 | gibi | I have a grenade job running on master as we speak so we will see if the problem is still exists | |
| 10:53:55 | gibi | https://zuul.opendev.org/t/openstack/status#810909 | |
| 10:53:58 | lyarwood | ack thanks | |
| 10:54:23 | lyarwood | it failed right at the end after the upgrade to master for me | |
| 10:54:33 | lyarwood | likely before that revert landed | |
| 11:10:16 | gibi | lyarwood: https://zuul.opendev.org/t/openstack/status#810909 still fails with trunks 404 :/ so there is something else | |
| 11:11:15 | gibi | so far I only found one small change that is in openstack/grenade switching to test upgrade from xena -> master instead of wallaby -> master | |
| 11:48:27 | gibi | lyarwood: quick summary. It seems devstack decides to generate [network-feature-enabled]api_extensions = all to the tempest.conf but the trunk extenison is not in neutrons [DEFAULT]service_plugins config | |
| 11:48:43 | lyarwood | nice | |
| 11:49:06 | gibi | on stable/wallaby the [network-feature-enabled]api_extensions did listed extensions explicity, but did not list the trunk extension so greande skips the trunk testing | |
| 11:50:00 | gibi | on master nova-next has api_extensions=all but trunk is in the service_plugins list | |
| 11:50:08 | gibi | so trunk passing there | |
| 12:01:32 | lyarwood | enable_service neutron-trunk is missing from the job | |
| 12:01:39 | lyarwood | I think that's it | |
| 12:02:09 | gibi | lyarwood: but how that got dropped? | |
| 12:02:40 | gibi | we did not run trunk on stable/wallaby in grenade but now on master we try | |
| 12:03:39 | gibi | I think the problem is not that we forgot to add neutron-trunk on master, but that somehow the trunk testing got enabled in grenade on master | |
| 12:03:49 | gibi | implicitly | |
| 12:04:12 | gibi | but sure one way to fix this is to add neutron-trunk to grenade on master | |
| 12:04:59 | lyarwood | https://github.com/openstack/tempest/blob/ae41052a51f5dbb748eb6bf4f23e9145853f4639/tempest/api/compute/admin/test_live_migration.py#L252 yeah there's a skip on the test but I guess the tempest conf has changed? | |
| 12:05:34 | gibi | yepp | |
| 12:05:56 | gibi | in the tempest conf [network-feature-enabled]api_extensions instructs tempest what to run or skip | |
| 12:06:22 | gibi | and that value change from an explicit list without trunk to a simple 'all' value | |
| 12:07:47 | gibi | in our grenade testing | |
| 12:07:59 | gibi | and actually it seems in every testing on mater | |
| 12:08:00 | gibi | master | |
| 12:08:21 | gibi | in nova-next we also get 'all' but in nova-next neutron-trunk is enabled | |
| 12:08:26 | gibi | so the test passes there | |
| 12:08:58 | gibi | bottom line our grenade tempest config got inconsistent as it is configured to run trunk tests without neutron-trunk | |
| 12:10:55 | lyarwood | kk I'll add the service in the parent grenade job | |
| 12:14:13 | gibi | OK, lets unblock the gate with that change | |
| 12:14:35 | gibi | is it enough to merge that to master or we need to backport that to stable/xena as well? | |
| 12:15:36 | lyarwood | that should be enough on master | |
| 12:15:41 | gibi | ack | |
| 12:15:42 | lyarwood | it's the zuul job itself after all | |
| 12:15:55 | lyarwood | https://review.opendev.org/c/openstack/grenade/+/811117 | |
| 12:17:17 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM testing grenade neutron-trunk fix https://review.opendev.org/c/openstack/nova/+/811118 | |
| 12:17:56 | gibi | lyarwood: thanks. | |
| 12:31:18 | lyarwood | ttps://review.opendev.org/q/topic:bug/1943431 is ready for reviews if anyone has time | |
| 12:31:29 | lyarwood | https://review.opendev.org/q/topic:bug/1943431 even | |
| 12:39:46 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810911 | |
| 12:41:03 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810913 | |
| 12:47:04 | spatel | sean-k-mooney does Intel Corporation Ethernet 10G 2P X550-t Adapter support SRIOV ? | |
| 12:47:18 | spatel | its copper 10G | |
| 12:49:52 | opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810915 | |
| 12:50:06 | spatel | Its saying but i can't see in lspci -vv command out for any SRIOV reference. let me check in BIOS | |
| 13:08:23 | spatel | sean-k-mooney nevermind, after enable in BIOS > NIC > Config (enable:SRIOV) | |
| 13:08:47 | spatel | i can see SRIOV support in lspci -vv | |
| 14:00:22 | bauzas | gibi: +2d for https://review.opendev.org/c/openstack/nova/+/810461 but I'd like to first tell about this during tomorrow's meeting before merging it | |
| 14:00:55 | bauzas | it looks to me we have a consensus but I'd prefer to see if anyone has a concern | |
| 14:08:50 | gibi | bauzas: sure | |
| 14:09:28 | bauzas | (and I could miss one=) | |
| 14:10:29 | gibi | the third is the stable setuptools fix | |
| 14:10:31 | gibi | I guess :D | |
| 14:40:10 | opendevreview | David Vallee Delisle proposed openstack/nova master: rephrasing config description for num_pcie_ports in libvirt https://review.opendev.org/c/openstack/nova/+/811173 | |
| 15:11:18 | melwitt | gibi, dansmith: just wanted to check whether you wanted to look at the stable patch validation bypass script before approval: https://review.opendev.org/c/openstack/nova/+/810285 | |
| 15:12:09 | fungi | sean-k-mooney: melwitt: https://review.opendev.org/811181 is the errata update for ossa-2021-002, now that the new fix for stable/ussuri has merged. let me know if you notice anything obviously wrong, i'm planning to send it to mailing lists later today | |
| 15:15:03 | gibi | melwitt: looks good to me. I let you +A it | |
| 15:16:58 | melwitt | fungi: thanks, will look | |
| 15:17:03 | melwitt | gibi: thanks! | |
| 15:54:03 | bauzas | gibi: ++ that was it (the setuptools fix) | |
| 16:18:30 | melwitt | fungi: sean-k-mooney is off today fyi | |
| 16:25:49 | fungi | no worries, just figured i'd see if anyone from nova wanted to give it a once-over. i'm realitively sure it's fit for publication if not | |
| 16:26:01 | fungi | relatively | |
| 16:27:29 | fungi | thanks melwitt! | |
| 18:59:32 | melwitt | elodilles: please let me know which approach you might prefer on the python 2.7 unit test issue on the stable/train open redirect fix patch https://review.opendev.org/c/openstack/nova/+/806629/2/nova/tests/unit/console/test_websocketproxy.py#662 | |
| 19:41:37 | elodilles | melwitt: commented on the patch. (sounds good to me) | |
| 19:49:50 | melwitt | thanks elodilles | |
| 21:18:57 | clarkb | heads up that devstack jobs are failing nova-status upgrade check checks because "GET /placemen//" status: 404 len: 162 microversion: 1.0 seems to be the request it is generating. For example https://zuul.opendev.org/t/openstack/build/2b59223b1c8047929b967ad52fdaed19/log/controller/logs/screen-placement-api.txt#257 | |
| 21:32:21 | melwitt | clarkb: ack, looking | |
| 21:34:29 | melwitt | "/placemen//" well, that's really weird :/ | |
| 21:57:03 | clarkb | ya I can't see any updates to keystoneauth or nova taht would explain that | |
| 22:03:19 | gmann | nor on placement sie | |
| 22:03:20 | gmann | side | |
| 22:04:16 | melwitt | same 😑 still looking for where that's coming from. no updates to keystoneauth, nova, placement, keystone. config files look normal | |
| 22:09:13 | gmann | its all started failing since last hr | |
| 22:13:52 | clarkb | let me know if we want to hold a node then we can debug directly | |
| 22:14:03 | clarkb | though in theory a local devstack would exhibit the same behavior if you ahve one | |
| 22:16:10 | melwitt | true | |
| 22:16:38 | melwitt | guess I'll go ahead and try that since I've not found anything else | |
| 22:30:17 | gmann | I do not see any installed pkgs version difference in passing vs failing build | |
| 22:45:36 | melwitt | my local devstack doesn't even get that far: | |
| 22:45:38 | melwitt | Installing collected packages: soupsieve, waitress, sortedcontainers, pyroute2.core, beautifulsoup4, WebTest, simplejson, [...] | |
| 22:45:40 | melwitt | Attempting uninstall: simplejson | |
| 22:45:42 | melwitt | Found existing installation: simplejson 3.16.0 | |
| 22:45:44 | melwitt | ERROR: Cannot uninstall 'simplejson'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. | |
| 22:47:56 | clarkb | melwitt: the issue there is whatever system you ran devstack on already has simplejson installed via system packages and pip can't uninstall it because the system package is deficient | |
| 22:48:32 | clarkb | I'ev brought it up with zigo as a good thing to help drive in the debuntu world by converting those packages to setuptools instead of distutils. Until then you can manually uninstall python-simplejson/python3-simplejson if on ubuntu. I assume similar package names on fedora/centos | |
| 22:48:42 | melwitt | hm, ok. I just created a new ubuntu focal VM. haven't had this happene before | |
| 22:49:34 | melwitt | ok thanks | |
| 22:54:02 | gmann | i am also creating new 20.04 machine | |
| 22:58:01 | melwitt | I noticed someone mentioned in #openstack-qa earlier today that they were able to hit the same error locally https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2021-09-27.log.html#t2021-09-27T19:54:38 | |
| 22:58:12 | johnsom | I am hitting it local | |
| 22:58:24 | melwitt | so hopefully one of us will figure out what's wrong | |
| #openstack-nova - 2021-09-28 | |||
| 00:00:49 | gmann | I am checking with sync mode also in case async tasks are creating issue here | |