| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-12 | |||
| 17:09:59 | sean-k-mooney | ah right | |
| 17:11:38 | sean-k-mooney | error_notification = self.notifier.wait_for_versioned_notifications( | |
| 17:11:39 | sean-k-mooney | 'compute.exception')[0] | |
| 17:11:44 | sean-k-mooney | we are just takign the first error | |
| 17:11:45 | gibi | yepp | |
| 17:11:57 | gibi | I have to drop for today | |
| 17:12:03 | sean-k-mooney | could it be the order of the excption could change and we get more then one | |
| 17:12:08 | sean-k-mooney | ok | |
| 17:12:09 | gibi | I pushed a small patch with an extra log seeing what triggers the reschedule | |
| 17:12:26 | gibi | https://review.opendev.org/c/openstack/nova/+/813674 | |
| 17:12:38 | sean-k-mooney | im not sure there is a reschdule | |
| 17:12:48 | sean-k-mooney | but cool lets see what that shows | |
| 17:12:53 | sean-k-mooney | ill play with a bit locally | |
| 17:13:19 | gibi | this line in the stack trace File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2263, in _do_build_and_run_instance | |
| 17:13:26 | gibi | points to a reschedule for me | |
| 17:14:30 | gibi | anyhow leaving now. thanks for the shared thinking | |
| 17:14:35 | gibi | o/ | |
| 17:19:45 | sean-k-mooney | hum | |
| 17:19:48 | sean-k-mooney | ERROR: Cannot install jsonschema>=3.2.0, openstack-placement==1.0.0 and openstack-placement==1.1.0 because these package versions have conflicting dependencies. | |
| 17:19:50 | sean-k-mooney | The conflict is caused by: | |
| 17:19:52 | sean-k-mooney | The user requested jsonschema>=3.2.0 | |
| 17:19:54 | sean-k-mooney | openstack-placement 1.1.0 depends on jsonschema<3.0.0 and >=2.6.0 | |
| 17:19:56 | sean-k-mooney | The user requested jsonschema>=3.2.0 | |
| 17:19:58 | sean-k-mooney | openstack-placement 1.0.0 depends on jsonschema<3.0.0 and >=2.6.0 | |
| 17:20:00 | sean-k-mooney | The user requested (constraint) jsonschema===3.2.0 | |
| 17:20:11 | sean-k-mooney | that was form trying to run nova func test on master | |
| 17:20:17 | sean-k-mooney | it cant create the pip env | |
| 17:21:35 | sean-k-mooney | placmenet required >=3.2.0 | |
| 17:21:36 | sean-k-mooney | https://github.com/openstack/placement/blob/master/requirements.txt#L10 | |
| 17:23:00 | sean-k-mooney | which is the same as uc https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L547 | |
| 17:24:18 | sean-k-mooney | oh | |
| 17:24:30 | sean-k-mooney | this is the same suds-jurko issue | |
| 17:25:31 | sean-k-mooney | ok i just need to locally comment out oslo.vmware in our test-requirements.txt until that is fixed | |
| 17:33:17 | opendevreview | Ghanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/813679 | |
| 17:36:12 | opendevreview | Ghanshyam proposed openstack/placement master: Use 'placement-nova-functional-py38' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/813680 | |
| 18:11:13 | opendevreview | Ghanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/813679 | |
| 18:13:23 | opendevreview | Ghanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/813679 | |
| 19:06:01 | gmann | gibi: bauzas these fix the placement-nova functional job - https://review.opendev.org/q/topic:%22fix-placement-gate%22+(status:open%20OR%20status:merged) | |
| 19:08:30 | opendevreview | sean mooney proposed openstack/nova master: [WIP] adress intermitent failure of functional tests https://review.opendev.org/c/openstack/nova/+/813695 | |
| 19:09:15 | sean-k-mooney | gibi: i think ^ would fix https://bugs.launchpad.net/nova/+bug/1946339 | |
| 21:09:00 | simondodsley | How would I achieve full disaster recovery of nova instances from one OS cluster to another? I can replicate Cinder volumes to another cluster (sort of), but I can't find a way to replicate the nova instances? I know Stratoscale used to do something like this, but they are dead now. Any other ways to do this? | |
| 21:21:49 | melwitt | simondodsley: I can't answer your question, I'm sure there are a lot of ways to do it, but you might get some ideas from this project https://docs.openstack.org/freezer/latest/ this is/was the openstack project for disaster recovery. it hasn't had activity for the past year or so, it may no longer be maintained https://github.com/openstack/freezer-dr | |
| 21:53:46 | opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | |
| 22:04:50 | gmann | melwitt: dansmith please check these two to unblock the placement gate https://review.opendev.org/q/topic:%22fix-placement-gate%22+(status:open%20OR%20status:merged) | |
| 22:29:38 | melwitt | gmann: hm, not sure I understand the solution. it doesn't seem right to define any placement env in the nova repo? I also don't understand why the current setup is failing | |
| 22:31:01 | gmann | melwitt: as it is used in nova-placement job we can move the job definition also on nova side but that run only in placement | |
| 22:32:05 | melwitt | gmann: I don't understand why the current thing is no longer working, it was intended to be able to use the nova-tox-functional-py38 in other projects right? as a parent job? | |
| 22:32:32 | melwitt | why do we need to add nova-placement in the nova or placement repo now? | |
| 22:35:28 | melwitt | let me read the commit message and referenced zuul commit again | |
| 22:35:30 | gmann | melwitt: placement-nova-tox-functional-py38 job is only needed to skip the sample and db tests otherwise same as nova-tox-functional-py38 | |
| 22:35:52 | gmann | and I think these tests are skipped as they do not use placement_fixture | |
| 22:36:00 | melwitt | yeah, I see that ... trying to understand the bug and why we can't use the parent job as intended | |
| 22:37:17 | melwitt | so a recent commit added tox_extra_args back, apparently previously it wasn't being used even though it was defined in the placement .zuul.yaml I guess | |
| 22:37:29 | melwitt | and now that it's being used, there's a syntax error happening | |
| 22:37:39 | gmann | melwitt: it is now started failing because test regex to skip the test is defined in tox_extra_args which is now added in 'Get tox envlist config; tasks https://opendev.org/zuul/zuul-jobs/commit/c02c28a982da8d5a9e7b4ca38d30967f6cd1531d | |
| 22:37:48 | gmann | yes | |
| 22:38:29 | gmann | I do not think tox_extra_args should be used for tests regex formation instead we should define the test path/regex etc in tox env command itself | |
| 22:39:08 | gmann | and as tox env has to live in nova and job is defined in placement, we need these two repo changes to fix it | |
| 22:39:44 | melwitt | oh... at least to me it looks like it makes sense for the regex string to be an extra arg to tox | |
| 22:40:36 | gmann | but that fail when we take tox_extra_args in tox env with no tests run like this https://opendev.org/zuul/zuul-jobs/commit/c02c28a982da8d5a9e7b4ca38d30967f6cd1531d | |
| 22:41:16 | gmann | which is generic to add tox_extra_args because it can have more config things for tox | |
| 22:41:50 | gmann | melwitt: easy way is to just run nova-tox-functional-py38 and do not skip tests as it is periodic weekly job | |
| 22:42:21 | melwitt | sorry, I got lost at the "no tests run" and "skip tests" | |
| 22:42:34 | gmann | or define the placement-nova-tox-functional-py38 job in nova but use in placement only | |
| 22:45:23 | gmann | melwitt: no test run when tox is run to show config --showconfig that is where it is failing when tox_extra_args has the test regex string | |
| 22:45:25 | melwitt | ok, I see now the "name: Run tox without tests" that is what doesn't work with the regex as an extra arg I take it | |
| 22:46:51 | gmann | melwitt: here too when for cinfig https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml#L22 | |
| 22:47:26 | melwitt | yeah... I see now | |
| 22:48:11 | melwitt | thanks | |
| 22:50:08 | gmann | if having tox env but not used on nova is confusing then we can move job definition too in nova but run in placement only | |
| 22:50:52 | gmann | otherwise running nova-tox-functional-py38 itself is not so costly as this will be run periodic weekly only | |
| 22:52:55 | melwitt | I dunno if it's confusing to most, it was just unintuitive to me because normally we can keep things separate and inherit from one project to another but I see now why this isn't possible with the regex we need to use. none of the zuul tox role variables are appropriate for this | |
| 22:54:48 | gmann | yeah | |
| 22:55:29 | melwitt | I think the way you have proposed probably makes the most sense | |
| 22:55:33 | melwitt | (what you have already proposed patches) | |
| 22:56:52 | gmann | ok, I added about which job is using this tox in its description to make it clear for future | |
| 22:57:46 | melwitt | yeah that is good ++ | |
| 23:01:38 | gmann | ok | |
| 23:07:51 | dansmith | gmann: each tox env is just a new target that does what we want the other job to do, is that right? | |
| 23:08:08 | dansmith | meaning, nova defines a placement env so placement can run nova tests "the placement way" ? | |
| 23:08:37 | dansmith | I think it would be super less confusing if the tox env name didn't have the other project in it, but described the difference, like "functional-all-but-foo-tests" or something | |
| 23:09:01 | melwitt | that's a good point ^ | |
| 23:09:04 | gmann | dansmith: ok | |
| 23:09:12 | gmann | placement does not need to be in name itself | |
| 23:10:37 | dansmith | I mean, just MHO, but.. seems more intuitive to me | |
| 23:11:10 | gmann | how about this 'functional-skip-sample-db-tests' | |
| 23:11:26 | melwitt | yeah maybe like functional-without-sample-tests-py38 or something | |
| 23:11:28 | melwitt | I agree, I think it would be more intuitive | |
| 23:11:42 | dansmith | cha | |
| 23:12:18 | melwitt | gmann: that sounds good, add -py38 at the end though right since it's 3.8 only? | |
| 23:12:19 | gmann | functional-without-sample-db-tests ? as it skip db test too and remove py38 thing as it can lengthy the name | |
| 23:12:25 | melwitt | oh | |
| 23:12:50 | gmann | melwitt: i was thinking just to run it on available py version as goal is not test py version in this job ? | |
| 23:13:14 | gmann | and we do not need to rename when we move to py39 or so | |
| 23:13:19 | melwitt | gmann: yeah true | |
| 23:13:31 | melwitt | yeah that makes sense | |
| 23:13:55 | gmann | ok, let me rename it. | |
| 23:14:07 | melwitt | k. will re +2 them after | |
| 23:14:14 | gmann | dansmith: melwitt do you think to move job definition also to nova side or it is ok? | |
| 23:15:40 | melwitt | gmann: I think the job def in placement is better, personally | |