| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-14 | |||
| 14:48:52 | mriedem | efried: was there anything you wanted to go over? my thing in open discussion is in the ML and i'm just going to do a quick PoC I think | |
| 14:49:02 | mriedem | there were a few stable branch releases | |
| 14:49:08 | mriedem | and elod was added to the nova-stable-maint team | |
| 14:52:22 | efried | mriedem: The other open (which luyao hadn't put on the agenda as of meeting start time) was whether the LM bits of vPMEM need a BP/spec or not. What are your thoughts? | |
| 14:55:20 | mriedem | my gut reaction is vPMEM hasn't even baked to the point of thinking about supporting live migration | |
| 14:55:40 | mriedem | i.e. i don't think we even support live migration with vgpus do we? | |
| 14:55:48 | mriedem | and people might actually be using that by now | |
| 14:56:12 | mriedem | vpmem and live migration is likely complicated that a short spec probably doesn't hurt | |
| 14:56:20 | mriedem | *complicated enough | |
| 14:56:45 | mriedem | stephenfin: i commented on https://review.opendev.org/#/c/692374/ | |
| 14:57:02 | mriedem | tl;dr i don't think USE_PYTHON3=True is making it to the devstack-plugin-ceph run when the post-test script runs it | |
| 14:57:36 | mriedem | stephenfin: also, in trying to convert the nova-live-migration job to zuulv3 here https://review.opendev.org/#/c/693364/ oi | |
| 14:57:53 | mriedem | i'm coming to the conclusion that it would be a ton simpler to just split the nova-live-migration job in two, | |
| 14:58:06 | mriedem | one for block migration and local disk and one with ceph + shared disk | |
| 14:58:14 | efried | mriedem: Pretty sure the impl is already done, it just didn't make Train. Anyway, I agree it could do with a short spec, as I think it needs new object fields, so we should at least be able to bikeshed the names of those. <== luyao | |
| 14:58:35 | mriedem | efried: is the vpmem 3rd party ci stable? | |
| 14:58:46 | mriedem | and if it's stable, can they run it multinode to test live migration? | |
| 14:58:50 | efried | dunno. alex_xu luyao ^ ? | |
| 14:58:51 | mriedem | at least periodically? | |
| 14:59:18 | efried | I thought the CI was stable at the end of train, for single node. Don't know the status of LM CI. | |
| 14:59:38 | mriedem | i just don't have a ton of energy to invest in building feature functionality on top of a thing that i don't know if anyone is using it | |
| 14:59:50 | mriedem | like SEV in that way... | |
| 15:00:21 | mriedem | having said all that, i think dansmith would be totally on board with building in live migration support for vpmems, that's right up his alley | |
| 15:01:01 | efried | I think of it less as "tack a feature onto this thing" and more as "piece of the puzzle that didn't quite get finished with the rest". | |
| 15:02:16 | efried | oh, maybe I'm wrong. I see the patch wasn't proposed until Oct. | |
| 15:02:32 | mriedem | so looking at a recent pmem ci run http://52.27.155.124/79/694179/2/check/pmem-tempest-plugin-filtered/1123b99/job-output.txt | |
| 15:02:36 | mriedem | it runs 3 tests in less than a minute | |
| 15:02:43 | mriedem | the entire job takes about 30 minutes | |
| 15:03:05 | mriedem | surely there is more testing that could be done with operations on a server with pmems, though i'm not sure what test_server_basic_ops does | |
| 15:03:24 | mriedem | e.g. if resize works, cold migration should work, so maybe first steps are making that ci job multinode and testing cold migrate | |
| 15:04:05 | sean-k-mooney | looking at http://ciwatch.mmedvede.net/project?project=nova it seams somewhat statble although it is missing some patches | |
| 15:04:41 | mriedem | ok here is the basic ops test https://github.com/LuyaoZhong/vpmem-tempest-plugin/blob/master/vpmem_tempest_plugin/tests/scenario/test_server_basic_ops.py#L127 | |
| 15:04:56 | mriedem | create a server, ssh into it, delete the server... | |
| 15:05:03 | stephenfin | mriedem: https://review.opendev.org/#/c/694281/1 | |
| 15:05:03 | mriedem | so reboot + pmem isn't being tested | |
| 15:05:06 | mriedem | shelve/unshelve | |
| 15:05:16 | stephenfin | I think that fixes the Python 3 issues, right? | |
| 15:05:23 | stephenfin | Or have I missed something obvious? | |
| 15:06:16 | sean-k-mooney | mriedem: i suspect pause/unpause and suspend/resume are also missing | |
| 15:06:37 | stephenfin | obviously converting things to zuul v3 native would be better though, of course, but maybe I should squash that back so we can continue working on the Python 3 only'ification in parallel | |
| 15:07:17 | mriedem | stephenfin: looks like it does yeah, py3 is being used here https://zuul.opendev.org/t/openstack/build/2850353a84da4ec08b4f7cda7a08a781/log/logs/subnode-2/screen-n-cpu.txt.gz#3 | |
| 15:07:20 | sean-k-mooney | mriedem: they have a resize test https://github.com/LuyaoZhong/vpmem-tempest-plugin/blob/master/vpmem_tempest_plugin/tests/scenario/test_server_pmem_ops.py#L105 | |
| 15:07:27 | mriedem | sean-k-mooney: that would be resize to the same host though | |
| 15:07:32 | stephenfin | \o/ | |
| 15:07:38 | sean-k-mooney | yes | |
| 15:07:54 | mriedem | stephenfin: so you want to squash that into https://review.opendev.org/#/c/692374/ ? | |
| 15:08:05 | stephenfin | yeah | |
| 15:08:08 | mriedem | go for it | |
| 15:08:08 | sean-k-mooney | but if resize on same host work then they should be able to resize to a different host no? if it was run multinode | |
| 15:08:13 | stephenfin | though maybe I should fix the devstack plugin first? | |
| 15:08:28 | mriedem | stephenfin: mostly the cinder core team | |
| 15:08:41 | mriedem | https://review.opendev.org/#/admin/groups/1196,members | |
| 15:08:45 | stephenfin | okay, they'll want this too so | |
| 15:08:52 | stephenfin | I'll fix that and add depends-on | |
| 15:09:09 | luyao | mriedem, efried : VPMEM CI is stable, but we haven't LM test yet. | |
| 15:09:30 | mriedem | the zuulv3 conversion of nova-live-migration gets super wonky with that post-test-script because the script assumes devstack-gate and still requires that for the nova-multinode-grenade job | |
| 15:09:57 | mriedem | so that's kind of why i'm tempted to convert nova-live-migration to zuulv3 and split it into two jobs, one with -ceph | |
| 15:10:04 | efried | luyao: I agree with mriedem that it would be good to expand the CI with more scenarios testing the existing code before adding LM support, see scrollback. | |
| 15:10:23 | mriedem | luyao: could you start by making the pmem ci job multinode and adding a test for cold migrate? | |
| 15:10:43 | mriedem | and shelve/unshelve | |
| 15:10:48 | stephenfin | Yeah, I spent the bones of a day pre-PTO trying to convert it and got nowhere so I'm glad you're working on it 😅 | |
| 15:10:57 | stephenfin | Splitting it up makes sense to me | |
| 15:11:08 | mriedem | it was pretty easy until i got to the script | |
| 15:11:10 | mriedem | which is a nightmare | |
| 15:12:32 | luyao | mriedem, efried : OK, I will start preparing more CI tests | |
| 15:13:42 | efried | luyao: So I still think because of the RPC (and therefore upgrade) impacts, we should definitely have a short spec for LM. In that spec, we should state that moving forward with LM will be contingent on first demonstrating CI coverage for $stuff above. | |
| 15:14:32 | luyao | efried: get it, thanks | |
| 15:15:23 | huaqiang | sean-k-mooney: about the per-vm-pci-NUMA-policy spec, I want to confirm that it defines a policy applied to all PCI devices for whole VM, right? | |
| 15:15:26 | gibi_off | eandersson: hi! have you opened a bug for the false error log in case of https://github.com/openstack/nova/commit/a5269012a3b442a9e4055a7d523faff45f105f2b#diff-77f9348ab09642ba46409b6828af4af0R1327 being hit on an empty compute? | |
| 15:15:34 | stephenfin | Dumb question, but does Ubuntu allow you to install e.g. 'python-foo' and 'python3-foo' side-by-side without them stomping on each other? | |
| 15:15:49 | efried | stephenfin: shore | |
| 15:15:56 | stephenfin | kewl | |
| 15:16:02 | sean-k-mooney | huaqiang: yes | |
| 15:16:16 | sean-k-mooney | huaqiang: and it takes precidnce over the alias in the config | |
| 15:16:48 | sean-k-mooney | so it applies to both neutron sriov ports and to flavor based pci passthough | |
| 15:17:10 | huaqiang | sean-k-mooney: then it is impossible to set three and more than three NUMA policies to PCI devices in a VM | |
| 15:17:57 | huaqiang | and we have three PCI numa policies in total | |
| 15:17:59 | sean-k-mooney | if you mean you cant have different policies for different devices correct | |
| 15:18:35 | huaqiang | yes, that is I want to confirm | |
| 15:18:40 | sean-k-mooney | that is why its titled vm-scoped-sriov-numa-affinity | |
| 15:19:17 | sean-k-mooney | if we want to do interface scoped sriov polices we still can as a seperate qos policy on the neuton port | |
| 15:19:39 | sean-k-mooney | i would like to enable that too at some point but after this initall effort is done once we see if its required | |
| 15:20:23 | luyao | efried, mriedem: I have another question for "track error mirations and orphans in resource tracker", I think it is a bug, which have some effects on vpmem feature. Do I need put all vpmem live migration stuff based on the bug fixed. I'm not sure how to address this bug at the moment. Could you help me figure it out, I sent an emaila few days ago. | |
| 15:22:44 | sean-k-mooney | luyao: i think those are two different topics | |
| 15:22:53 | luyao | I put my solution on this etherpad. I hope it is clear for you. We can discuss on the etherpad. https://etherpad.openstack.org/p/track-err-migr-and-orphans-in-RT | |
| 15:23:01 | efried | luyao: I'll try to look, but it sounds daunting. If you think there's a bug (vpmem notwithstanding) we would welcome a patch to fix it. | |
| 15:24:32 | sean-k-mooney | efried: part of it is a design choice. which is been adressed in a bugfix already | |
| 15:25:09 | huaqiang | sean-k-mooney: Got. thanks | |
| 15:25:16 | sean-k-mooney | specificly there is a patch to reap orpahanded vms that have a nova metada in there domain xml but were deleted in the db | |
| 15:25:22 | luyao | efried, sean-k-mooney : so I can do the vpmem live migration and bug fix parallelly,right? | |
| 15:25:33 | sean-k-mooney | as an extention to the existing periodic task | |
| 15:25:42 | efried | mriedem: https://review.opendev.org/#/c/694248/ FYI (wait to recheck patches failing requirements check until ^ merges) | |
| 15:25:52 | efried | (per smcginnis) | |
| 15:26:14 | sean-k-mooney | luyao: provided you dont need to modify the same code sure even then you can factor out the common code into a shared patch | |
| 15:26:33 | sean-k-mooney | luyao: from a review persepective it will be simpler if they are seperate | |
| 15:27:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: docs: update SUSPENDED server status wrt supported drivers https://review.opendev.org/694329 | |
| 15:27:41 | mriedem | efried: i'm not sure what you're showing me | |
| 15:28:03 | mriedem | is that related to why my sqla-migrate change failed? | |
| 15:28:07 | efried | mriedem: the reqs job is broken until that merges. We've got a couple patches in the pipe that keep failing on reqs. yes. | |
| 15:28:25 | mriedem | ack, thanks | |