| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-12-10 | |||
| 17:35:32 | gmann | gibi: sean-k-mooney did not get why it started failing now? is there any recent change triggered this? | |
| 17:43:10 | opendevreview | Ghanshyam proposed openstack/nova master: DNM: testing nova-ceph-multistore fix https://review.opendev.org/c/openstack/nova/+/821435 | |
| 17:47:51 | sean-k-mooney | gmann: not that i know if i suspect either neutron started requrieing a new version of python3-logging or there was a cpeh package update upstream | |
| 17:54:26 | gmann | i can see only in l-c requirements which should not be installed by devstack | |
| 17:56:46 | sean-k-mooney | ya i was checking on codeasearch | |
| 17:56:52 | sean-k-mooney | not sure what is pulling it in to be honest | |
| 17:59:30 | sean-k-mooney | sean@cloud:~$ apt-cache rdepends python3-logutils | |
| 17:59:32 | sean-k-mooney | python3-logutils | |
| 17:59:34 | sean-k-mooney | Reverse Depends: | |
| 17:59:36 | sean-k-mooney | python3-pecan | |
| 17:59:38 | sean-k-mooney | changeme | |
| 17:59:47 | sean-k-mooney | so its comiy for either python3-pecan or changeme | |
| 17:59:56 | sean-k-mooney | likely python3-pecan | |
| 18:02:10 | sean-k-mooney | ya so thye had a release 2 days ago | |
| 18:02:12 | sean-k-mooney | https://pypi.org/project/pecan/ | |
| 18:02:17 | sean-k-mooney | and its used by neturon | |
| 18:02:57 | sean-k-mooney | gmann: and we udpated the upper constratints this morning https://github.com/openstack/requirements/commit/e74f91e250d127ff825dad5219527d62efaeb740 | |
| 18:03:00 | gmann | humm, it was there in pecan 1.4.1 which constraints updated in requirement yesterday https://github.com/pecan/pecan/blob/1.4.1/requirements.txt | |
| 18:03:10 | sean-k-mooney | yep | |
| 18:03:16 | sean-k-mooney | so that is likely what broke it | |
| 18:03:22 | gmann | yeah but there is no change in previous version of pecan on logutils | |
| 18:03:35 | sean-k-mooney | https://github.com/pecan/pecan/blob/master/requirements.txt#L6 | |
| 18:03:43 | sean-k-mooney | the require logutile >=3 | |
| 18:03:58 | gmann | yeah that is there in old version too https://github.com/pecan/pecan/blob/1.3.3/requirements.txt | |
| 18:04:07 | sean-k-mooney | ya 8 years | |
| 18:04:47 | sean-k-mooney | im not sure but that is the best i can come up with the version that is shipped in ubuntu is 0.3.4 i think | |
| 18:05:09 | sean-k-mooney | 0.3.3-7 | |
| 18:05:23 | sean-k-mooney | so ghatt should work | |
| 18:05:33 | gmann | I am trying to check the passing nova-ceph-multistore job and it was not pulled there somwhoe https://zuul.openstack.org/build/590c5996ca1b402486bfe1c7e1d08535/log/job-output.txt | |
| 18:07:29 | sean-k-mooney | its likely a ubuntu or cpeh packageing change | |
| 18:07:39 | sean-k-mooney | to we use the disto ceph or ceph form ceph.com | |
| 18:08:32 | sean-k-mooney | oh | |
| 18:08:38 | sean-k-mooney | the ceph pcakages changed | |
| 18:08:40 | sean-k-mooney | CEPH_PACKAGES='ceph libnss3-tools python3-rados python3-rbd' | |
| 18:08:43 | sean-k-mooney | on passin g | |
| 18:10:34 | sean-k-mooney | actully they end up being the same | |
| 18:10:36 | sean-k-mooney | CEPH_PACKAGES='ceph libnss3-tools python3-rados python3-rbd' | |
| 18:10:51 | gmann | yeah its same | |
| 18:12:05 | sean-k-mooney | et:20 https://download.ceph.com/debian-pacific focal/main amd64 python3-rbd amd64 16.2.7-1focal [386 kB] | |
| 18:12:08 | sean-k-mooney | vs | |
| 18:12:23 | sean-k-mooney | 021-12-10 10:37:05.265864 | controller | Get:48 https://download.ceph.com/debian-pacific focal/main amd64 python3-rbd amd64 16.2.7-1focal [386 kB] | |
| 18:12:33 | sean-k-mooney | so i guess the same version of ceph too | |
| 18:21:58 | gmann | sean-k-mooney: commented about commit msg to add these info and then i am +2 as it is gate blocker https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/821396/comment/a2721f12_ea844e6b/ | |
| 18:22:28 | gmann | i cannot find root cause but it is clear it is pulling it recently so removing it is no issue | |
| 18:23:04 | gmann | I can try to ping other core if they are around | |
| 18:32:32 | sean-k-mooney | gibi: thanks yes ill spellcheck it with gramerly on monday | |
| 18:33:39 | sean-k-mooney | gmann: am if tis ok with you ill update teh commit on monday i was just about to leave for the day | |
| 18:34:40 | gmann | sean-k-mooney: ok, or i can update for you if it is ok? and try to get other core if they are around gouthamr or so. so that we can get gate unblocked | |
| 18:37:52 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova stable/wallaby: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/816883 | |
| 20:06:49 | jamesbenson | Can anyone help with refstack tests failing due to additional-properties? | |
| 20:10:53 | jamesbenson | These are, for example this test: tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action | |
| 20:11:01 | jamesbenson | Gives me a: jsonschema.exceptions.ValidationError: Additional properties are not allowed ('updated_at' was unexpected) | |
| 20:12:38 | gmann | jamesbenson: hi | |
| 20:13:05 | gmann | jamesbenson: is it with upstream code or modified API? | |
| 20:13:48 | gmann | In tempest, we have strict validation on API response on additional-properties and t will fail if any API return any additional field in response without microversion or in downstream | |
| 20:15:09 | jamesbenson | okay, so I need to add the microversions to my tempest.conf then? Currently I don't have those. | |
| 20:15:39 | jamesbenson | It with a local deployment | |
| 20:17:42 | jamesbenson | @gmann https://gitlab.com/utsa-ics/osias We do regular testing and only ran into this issue with Victoria on. Ussuri wasn't giving any issues. But we haven't merged code for Victoria, Wallaby, or Xena yet because of these issues. | |
| 20:19:59 | gmann | jamesbenson: i see, please cap the max version for victoria in tempest conf as 2.87 max_microversion https://github.com/openstack/tempest/blob/93a902072fd9986f2bb660166552f37d9eb5bdbb/tempest/config.py#L381 | |
| 20:20:50 | jamesbenson | @gmann, Yeah, these are version locks we have: https://gitlab.com/utsa-ics/osias/-/blob/master/osias_variables.py#L67 | |
| 20:21:32 | jamesbenson | We need to specify both min and max version, correct? | |
| 20:22:13 | gmann | jamesbenson: min version can be None as Nova has not bumped the min version and its 2.1 only. min_microversion as None in tempest conf means nova 2.1 | |
| 20:22:27 | gmann | jamesbenson: so you are getting error with these cap version right? | |
| 20:23:43 | jamesbenson | @gmann, currently we don't have these caps used in our testing and are getting the errors. | |
| 20:24:20 | gmann | jamesbenson: so basically with your current configuration, tempest will request nova microversion 2.80 for victoria | |
| 20:24:42 | jamesbenson | We did use them at some point, but I think we were getting additional errors, so we stopped. But I'm running tests now, so I should know in a couple of hours. | |
| 20:24:52 | gmann | jamesbenson: make min_microversion as None and max_microversion as you have currently in https://gitlab.com/utsa-ics/osias/-/blob/master/osias_variables.py#L72 | |
| 20:24:56 | jamesbenson | Yes, these are what would get put into the tempest.conf file | |
| 20:25:17 | jamesbenson | okay, will do! | |
| 20:25:29 | gmann | jamesbenson: because tempest might fail with min_microversion as 2.80 because few tests might need modification on asserts/schema | |
| 20:26:00 | jamesbenson | gotcha, should I set them all to "None"? | |
| 20:26:07 | gmann | yeah | |
| 20:26:34 | jamesbenson | Xena I have a max of: 2.90 | |
| 20:26:59 | gmann | yeah, that's correct https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-xena | |
| 20:27:02 | jamesbenson | Does the tempest version or other things matter? | |
| 20:27:26 | jamesbenson | (And any thoughts on the project, feel free to share, we love input) | |
| 20:27:28 | gmann | jamesbenson: for victoria no. Tempest master works for ussuri - current openstack master | |
| 20:28:07 | gmann | that is what we test at upstream also, tempest master to test ussuri, victoria, wallaby, xena and current yoga | |
| 20:28:48 | jamesbenson | okay, cool. I'll update tempest u->x to 29.1.0 | |
| 20:29:52 | gmann | +1, please let me know if tempest fail in victoria as min_microversion=None and max_microversion='2.87' | |
| 20:30:58 | jamesbenson | sure will do | |
| 20:31:02 | jamesbenson | one other thing fails too | |
| 20:31:34 | jamesbenson | we use cirros as our test image, and gocubsgo works, but then some tests use random passwords, and those fail | |
| 20:31:39 | jamesbenson | Any ideas on that? | |
| 20:31:42 | jamesbenson | microversion issue too? | |
| 20:33:13 | gmann | jamesbenson: cirros is one we use in upstream too and password is configurable in tempest config. But tempest is not ready to run for all combination of micrversion. | |
| 20:33:38 | gmann | I will say try with min_microversion=None and max_microversion='2.87' and then we can see if they still fail then its tempest issue | |
| 20:34:29 | jamesbenson | ok, sounds good. These are our setups for tempest.conf | |
| 20:34:31 | jamesbenson | https://gitlab.com/utsa-ics/osias/-/blob/master/test_setup.sh | |
| 20:36:11 | gmann | yeah, that seems correct image_ssh_password but there might be some test issue, if you face issue ping me on #openstack-qa or file bug in tempest i can fix that | |
| 20:36:37 | jamesbenson | you are a life saver! Thank you! | |
| 20:37:13 | gmann | np! | |
| 20:39:53 | jamesbenson | I do have one re-occuring bug, it always happens in ussuri up in the tearDownClass (tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON). Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 5be11612-2a84-4723-b683-3cf06e3567d4: One or more ports have an IP allocation from this subnet.', 'detail': ''} | |
| 20:40:22 | jamesbenson | It can't destroy the subnet it creates.... | |
| 20:41:54 | gmann | jamesbenson: can you please log bug in tempest for that, I will check on Monday | |
| 20:42:08 | gmann | here https://bugs.launchpad.net/tempest | |
| 20:42:15 | jamesbenson | sure thing | |
| 20:42:33 | gmann | thanks | |
| 20:44:14 | jamesbenson | Any special logs you need or just the output from Refstack on that? | |
| 20:44:49 | gmann | you can add tempest log also | |