| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 18:58:17 | efried | gmann: What's the accepted way to add a specialized CI job to nova these days? | |
| 18:59:41 | sean-k-mooney | efried: are you asking about thrid party or first party | |
| 18:59:49 | efried | 1p, I would think | |
| 19:00:02 | efried | - build servers with custom flavors | |
| 19:00:02 | efried | - (compile and?) install some custom software | |
| 19:00:02 | efried | - (compile and?) install special versions of libvirt and qemu | |
| 19:00:02 | efried | - run barbican | |
| 19:00:02 | efried | Specifically, I'm going to need to work something up for vTPM. This is going to mean that the job will need to | |
| 19:00:03 | efried | - run some commands via ssh | |
| 19:00:07 | sean-k-mooney | you can just defien it in tree in the .zuul.yaml | |
| 19:00:25 | sean-k-mooney | the job name should start with nova | |
| 19:00:52 | sean-k-mooney | right | |
| 19:01:34 | sean-k-mooney | am its on my long todo list but i do plan to add a tempest jobs that does the libvirt/qemu compilation for that devstack plugin | |
| 19:02:00 | sean-k-mooney | that said next cycle ubuntu 20.04 should ship the ones you need | |
| 19:02:22 | sean-k-mooney | for the swtpm sorfware | |
| 19:02:27 | efried | I don't have until next cycle ;P | |
| 19:02:52 | sean-k-mooney | i would just use an ansible per-run playbook | |
| 19:02:53 | efried | I don't know that I care to run actual tempest | |
| 19:03:23 | efried | Unless the "build servers" and "run some commands" bits need to be done in a tempestuous framework of some kind? A plugin? | |
| 19:03:36 | efried | jroll: o/ | |
| 19:03:39 | sean-k-mooney | it could | |
| 19:03:57 | sean-k-mooney | but on the custom flaovr front have you looked at my dpdk job | |
| 19:04:13 | efried | my brain goes fuzzy when I see dpdk | |
| 19:04:19 | gmann | efried: you can use zuulv3 native base jobs from devstack if you do not want to run tempest | |
| 19:04:24 | sean-k-mooney | well dpdk is not important | |
| 19:04:44 | sean-k-mooney | but you can use a pre-run playbook to create a local.sh file which will create custome flavors | |
| 19:04:51 | sean-k-mooney | e.g. request vtpm | |
| 19:04:57 | efried | gmann: is there a "making a devstack zuulv3 job for dummies" guide somewhere? | |
| 19:05:03 | sean-k-mooney | then you can run standard tempest wtih those falvor to test your feature | |
| 19:05:17 | sean-k-mooney | or a subset of tempest | |
| 19:05:47 | gmann | efried: humm, not guide as such i know but i can link you some example from other project did like Tacker etc | |
| 19:05:56 | gmann | or job description | |
| 19:06:21 | efried | k. I don't trust nova's .zuul.yaml cause I never know when there's a legacy minefield I'm walking into. | |
| 19:07:01 | gmann | 'devstasck' can be used with minimum effort on specific jobs - https://github.com/openstack/devstack/blob/2e45f2c267c9ababdbdfc4c505b329398391c5f9/.zuul.yaml#L352 | |
| 19:07:35 | gmann | this is tacker multinode job for their functional testing - https://github.com/openstack/tacker/blob/bdb2d52b3a1b69c58cb2ac6f903380ab8a7bd973/.zuul.yaml#L28 | |
| 19:08:09 | efried | oh, so it's still kosher to use run.yaml things | |
| 19:08:12 | efried | that's a relief. | |
| 19:08:16 | sean-k-mooney | yes | |
| 19:08:30 | KeithMnemonic | melwitt. did this change break the patch? it is failing on a live migration thing now https://review.opendev.org/#/c/683008/6..7/nova/tests/unit/compute/test_compute.py | |
| 19:08:34 | gmann | multinode support is in all base jobs of devstack which is based on nodeset used on job | |
| 19:08:36 | sean-k-mooney | by default we leave it out in most of our jobs because its coming from the parent job | |
| 19:09:08 | melwitt | KeithMnemonic: lol no, that's a unit test. the live migration job is failing. we might need to rebase. I'm looking at the failure now | |
| 19:09:38 | efried | melwitt: if it's PyYAML, we're just waiting for https://review.opendev.org/#/c/703735/ to merge | |
| 19:09:52 | melwitt | unit tests run in the openstack-tox-py27 and openstack-tox-py35 jobs | |
| 19:10:06 | KeithMnemonic | ok thank you | |
| 19:10:57 | melwitt | efried: this is stable/pike so I doubt it right? the PyYAML is a new thing? | |
| 19:11:11 | efried | oh | |
| 19:11:17 | efried | I don't know. Is devstack branchless or something? | |
| 19:11:25 | melwitt | gah gzipped logs again | |
| 19:11:25 | sean-k-mooney | efried: here is an example job i need to go update https://review.opendev.org/#/c/679656/12 but it show you how to create a custom job with custom flavor and run standard tempest test to validate thing | |
| 19:11:33 | sean-k-mooney | efried: no devstack is branched | |
| 19:11:43 | melwitt | efried: devstack has branches but tempest is branchless | |
| 19:11:53 | sean-k-mooney | yes ^ | |
| 19:12:21 | efried | melwitt: okay, that failure looks different, ignore me. | |
| 19:12:40 | melwitt | how are these old crusty jobs having the friggin gzipped logs, dang | |
| 19:12:59 | efried | thanks sean-k-mooney | |
| 19:13:29 | efried | oh, yeah melwitt, that's going to continue to be a problem I guess, unless we plan to migrate legacy jobs to zv3 on stable. Much sad face. | |
| 19:13:44 | efried | we may have to figure out how to get that turned off in the job itself for stable. | |
| 19:13:45 | melwitt | oh geeez :''''''( | |
| 19:13:51 | sean-k-mooney | am well it should not be | |
| 19:13:59 | sean-k-mooney | oh actully | |
| 19:14:05 | efried | I thought it was an infra-level thing. | |
| 19:14:11 | sean-k-mooney | ya it is | |
| 19:14:37 | efried | so, if we're going to have to figure that out anyway, I guess we might as well do it on master, asap, so we can stop being frustrated at least partially. | |
| 19:14:39 | sean-k-mooney | so we need to modify devstack-gate to stop compressing the logs internally | |
| 19:14:46 | sean-k-mooney | and backport that to stable branches | |
| 19:15:18 | melwitt | isn't that what you did though? it didn't stop the compression for legacy jobs on master | |
| 19:15:30 | melwitt | so even if we backport it, it won't help right? | |
| 19:15:33 | sean-k-mooney | my fix fixed non legacy jobs | |
| 19:16:31 | sean-k-mooney | i intended it to fix both but it seams that devstack gate is compressing them somewhere else too | |
| 19:16:36 | sean-k-mooney | or the legacy jobs are | |
| 19:16:43 | sean-k-mooney | it might not be in devstack gate | |
| 19:16:54 | melwitt | got it | |
| 19:17:38 | melwitt | ok, now to find where this thing is failing LM | |
| 19:18:41 | sean-k-mooney | i have been doing " curl <log url> | zcat | lnav -q" | |
| 19:19:45 | melwitt | that's helpful, thanks | |
| 19:21:56 | sean-k-mooney | melwitt: artom was say you can also to "curl <url> | zless" but i like using lnav to browse logs | |
| 19:23:13 | melwitt | I still haven't gotten around to trying lnav so I'll use zless for now | |
| 19:23:47 | sean-k-mooney | ya if you do try it by default lnav dumps all the logs to std out so the -q is there to stop that | |
| 19:24:16 | melwitt | \:| [instance: 5bca844b-4ca6-4c63-a579-8f00316da019] Instance spawn was interrupted before instance_claim, setting instance to ERROR state | |
| 19:24:51 | sean-k-mooney | am that is new | |
| 19:24:59 | melwitt | yeah, it's new to me | |
| 19:25:18 | sean-k-mooney | does that mean qemu crashed or something? | |
| 19:25:25 | melwitt | no idea | |
| 19:25:51 | sean-k-mooney | well that makes two of us | |
| 19:26:34 | melwitt | I don't expect it's related to the patch, and the patch has been sitting around forever so I wouldn't be surprised if it needs a rebase by now. but the nova-live-migration job is failing pretty consistently on it and so far, for unknown reasons | |
| 19:26:58 | melwitt | rechecked it like 3 times and always fail on nova-live-migration | |
| 19:27:59 | sean-k-mooney | i can take a look at it a bit later. what is the review? | |
| 19:28:27 | sean-k-mooney | once i fix the osp 15 backport im doing ill take a quick look before moveing to osp 13 | |
| 19:28:30 | melwitt | sean-k-mooney: it's this one https://review.opendev.org/683008 | |
| 19:28:56 | sean-k-mooney | cool i have it open. | |
| 19:28:58 | melwitt | note that it is very old, not rebased since september | |
| 19:29:00 | artom | sean-k-mooney, huh, that's the same thing we saw in whitebox | |
| 19:29:25 | artom | We thought it was because we were restarting nova-compute to change configs | |
| 19:29:31 | sean-k-mooney | melwitt: if you do a rebase via the ui i should not hurt | |
| 19:29:39 | sean-k-mooney | but i dont think that is the issue | |
| 19:29:53 | sean-k-mooney | zuul will do a merge with master so it should basicaly be the same | |
| 19:29:56 | melwitt | yeah, probably worth it to try. it's gonna have to be rebased to merge anyway | |
| 19:30:06 | melwitt | let me do it now | |
| 19:30:39 | sean-k-mooney | well its more or less the same as a recheck and will avoid the merge commit so it wont hurt | |