| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 18:40:22 | sean-k-mooney | i think you still being working and only at 18:40 is enough pressue to make you fix anything | |
| 18:41:01 | sean-k-mooney | /only/online/ | |
| 18:44:52 | stephenfin | heh, yeah, I'll be done shortly | |
| 18:45:38 | artom | stephenfin, oh, I remember where I blocked - if we update the request spec with the new numa_topology, but the resize fails in one of a myriad of ways, and we need to revert it | |
| 18:46:04 | artom | IIRC we talked with dansmith about not persisting the numa_topology at all, and just making sure each request to the scheduler has the correct one | |
| 18:46:23 | stephenfin | artom: Just looked into that. FWICT, we don't revert the RequestSpec.flavor, let alone the RequestSpec.numa_topolology, in the case of a failure | |
| 18:46:36 | stephenfin | So I need another follow-up to do that | |
| 18:46:50 | artom | stephenfin, ah yeah, the flavor was in the same boat | |
| 18:48:35 | openstack | Launchpad bug 1831771 in OpenStack Compute (nova) "UnexpectedDeletingTaskStateError exception can leave traces of VIFs on host" [Medium,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 18:48:35 | sean-k-mooney | oh ye are talking about a different bug. i was refering to https://bugs.launchpad.net/nova/+bug/1831771 | |
| 18:49:57 | sean-k-mooney | sorry i saw you being ping about that downstream so assume that was the one you were refering too | |
| 18:52:58 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 18:52:59 | openstackgerrit | Eric Fried proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 18:58:17 | efried | gmann: What's the accepted way to add a specialized CI job to nova these days? | |
| 18:59:41 | sean-k-mooney | efried: are you asking about thrid party or first party | |
| 18:59:49 | efried | 1p, I would think | |
| 19:00:02 | efried | - build servers with custom flavors | |
| 19:00:02 | efried | - (compile and?) install some custom software | |
| 19:00:02 | efried | - (compile and?) install special versions of libvirt and qemu | |
| 19:00:02 | efried | - run barbican | |
| 19:00:02 | efried | Specifically, I'm going to need to work something up for vTPM. This is going to mean that the job will need to | |
| 19:00:03 | efried | - run some commands via ssh | |
| 19:00:07 | sean-k-mooney | you can just defien it in tree in the .zuul.yaml | |
| 19:00:25 | sean-k-mooney | the job name should start with nova | |
| 19:00:52 | sean-k-mooney | right | |
| 19:01:34 | sean-k-mooney | am its on my long todo list but i do plan to add a tempest jobs that does the libvirt/qemu compilation for that devstack plugin | |
| 19:02:00 | sean-k-mooney | that said next cycle ubuntu 20.04 should ship the ones you need | |
| 19:02:22 | sean-k-mooney | for the swtpm sorfware | |
| 19:02:27 | efried | I don't have until next cycle ;P | |
| 19:02:52 | sean-k-mooney | i would just use an ansible per-run playbook | |
| 19:02:53 | efried | I don't know that I care to run actual tempest | |
| 19:03:23 | efried | Unless the "build servers" and "run some commands" bits need to be done in a tempestuous framework of some kind? A plugin? | |
| 19:03:36 | efried | jroll: o/ | |
| 19:03:39 | sean-k-mooney | it could | |
| 19:03:57 | sean-k-mooney | but on the custom flaovr front have you looked at my dpdk job | |
| 19:04:13 | efried | my brain goes fuzzy when I see dpdk | |
| 19:04:19 | gmann | efried: you can use zuulv3 native base jobs from devstack if you do not want to run tempest | |
| 19:04:24 | sean-k-mooney | well dpdk is not important | |
| 19:04:44 | sean-k-mooney | but you can use a pre-run playbook to create a local.sh file which will create custome flavors | |
| 19:04:51 | sean-k-mooney | e.g. request vtpm | |
| 19:04:57 | efried | gmann: is there a "making a devstack zuulv3 job for dummies" guide somewhere? | |
| 19:05:03 | sean-k-mooney | then you can run standard tempest wtih those falvor to test your feature | |
| 19:05:17 | sean-k-mooney | or a subset of tempest | |
| 19:05:47 | gmann | efried: humm, not guide as such i know but i can link you some example from other project did like Tacker etc | |
| 19:05:56 | gmann | or job description | |
| 19:06:21 | efried | k. I don't trust nova's .zuul.yaml cause I never know when there's a legacy minefield I'm walking into. | |
| 19:07:01 | gmann | 'devstasck' can be used with minimum effort on specific jobs - https://github.com/openstack/devstack/blob/2e45f2c267c9ababdbdfc4c505b329398391c5f9/.zuul.yaml#L352 | |
| 19:07:35 | gmann | this is tacker multinode job for their functional testing - https://github.com/openstack/tacker/blob/bdb2d52b3a1b69c58cb2ac6f903380ab8a7bd973/.zuul.yaml#L28 | |
| 19:08:09 | efried | oh, so it's still kosher to use run.yaml things | |
| 19:08:12 | efried | that's a relief. | |
| 19:08:16 | sean-k-mooney | yes | |
| 19:08:30 | KeithMnemonic | melwitt. did this change break the patch? it is failing on a live migration thing now https://review.opendev.org/#/c/683008/6..7/nova/tests/unit/compute/test_compute.py | |
| 19:08:34 | gmann | multinode support is in all base jobs of devstack which is based on nodeset used on job | |
| 19:08:36 | sean-k-mooney | by default we leave it out in most of our jobs because its coming from the parent job | |
| 19:09:08 | melwitt | KeithMnemonic: lol no, that's a unit test. the live migration job is failing. we might need to rebase. I'm looking at the failure now | |
| 19:09:38 | efried | melwitt: if it's PyYAML, we're just waiting for https://review.opendev.org/#/c/703735/ to merge | |
| 19:09:52 | melwitt | unit tests run in the openstack-tox-py27 and openstack-tox-py35 jobs | |
| 19:10:06 | KeithMnemonic | ok thank you | |
| 19:10:57 | melwitt | efried: this is stable/pike so I doubt it right? the PyYAML is a new thing? | |
| 19:11:11 | efried | oh | |
| 19:11:17 | efried | I don't know. Is devstack branchless or something? | |
| 19:11:25 | melwitt | gah gzipped logs again | |
| 19:11:25 | sean-k-mooney | efried: here is an example job i need to go update https://review.opendev.org/#/c/679656/12 but it show you how to create a custom job with custom flavor and run standard tempest test to validate thing | |
| 19:11:33 | sean-k-mooney | efried: no devstack is branched | |
| 19:11:43 | melwitt | efried: devstack has branches but tempest is branchless | |
| 19:11:53 | sean-k-mooney | yes ^ | |
| 19:12:21 | efried | melwitt: okay, that failure looks different, ignore me. | |
| 19:12:40 | melwitt | how are these old crusty jobs having the friggin gzipped logs, dang | |
| 19:12:59 | efried | thanks sean-k-mooney | |
| 19:13:29 | efried | oh, yeah melwitt, that's going to continue to be a problem I guess, unless we plan to migrate legacy jobs to zv3 on stable. Much sad face. | |
| 19:13:44 | efried | we may have to figure out how to get that turned off in the job itself for stable. | |
| 19:13:45 | melwitt | oh geeez :''''''( | |
| 19:13:51 | sean-k-mooney | am well it should not be | |
| 19:13:59 | sean-k-mooney | oh actully | |
| 19:14:05 | efried | I thought it was an infra-level thing. | |
| 19:14:11 | sean-k-mooney | ya it is | |
| 19:14:37 | efried | so, if we're going to have to figure that out anyway, I guess we might as well do it on master, asap, so we can stop being frustrated at least partially. | |
| 19:14:39 | sean-k-mooney | so we need to modify devstack-gate to stop compressing the logs internally | |
| 19:14:46 | sean-k-mooney | and backport that to stable branches | |
| 19:15:18 | melwitt | isn't that what you did though? it didn't stop the compression for legacy jobs on master | |
| 19:15:30 | melwitt | so even if we backport it, it won't help right? | |
| 19:15:33 | sean-k-mooney | my fix fixed non legacy jobs | |
| 19:16:31 | sean-k-mooney | i intended it to fix both but it seams that devstack gate is compressing them somewhere else too | |
| 19:16:36 | sean-k-mooney | or the legacy jobs are | |
| 19:16:43 | sean-k-mooney | it might not be in devstack gate | |
| 19:16:54 | melwitt | got it | |
| 19:17:38 | melwitt | ok, now to find where this thing is failing LM | |
| 19:18:41 | sean-k-mooney | i have been doing " curl <log url> | zcat | lnav -q" | |
| 19:19:45 | melwitt | that's helpful, thanks | |
| 19:21:56 | sean-k-mooney | melwitt: artom was say you can also to "curl <url> | zless" but i like using lnav to browse logs | |
| 19:23:13 | melwitt | I still haven't gotten around to trying lnav so I'll use zless for now | |
| 19:23:47 | sean-k-mooney | ya if you do try it by default lnav dumps all the logs to std out so the -q is there to stop that | |
| 19:24:16 | melwitt | \:| [instance: 5bca844b-4ca6-4c63-a579-8f00316da019] Instance spawn was interrupted before instance_claim, setting instance to ERROR state | |
| 19:24:51 | sean-k-mooney | am that is new | |
| 19:24:59 | melwitt | yeah, it's new to me | |
| 19:25:18 | sean-k-mooney | does that mean qemu crashed or something? | |
| 19:25:25 | melwitt | no idea | |
| 19:25:51 | sean-k-mooney | well that makes two of us | |
| 19:26:34 | melwitt | I don't expect it's related to the patch, and the patch has been sitting around forever so I wouldn't be surprised if it needs a rebase by now. but the nova-live-migration job is failing pretty consistently on it and so far, for unknown reasons | |
| 19:26:58 | melwitt | rechecked it like 3 times and always fail on nova-live-migration | |