| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-25 | |||
| 13:55:47 | efried | cdent: ight, rubber-stamped. | |
| 13:57:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:03:49 | stephenfin | mriedem: Can we set different config options for fake services? e.g. test_compute1 has a different '[neutron_physnet_foo] numa_nodes' value from test_compute2 | |
| 14:04:24 | mriedem | ummm | |
| 14:04:30 | mriedem | i don't think so... | |
| 14:04:34 | mriedem | CONF is global | |
| 14:04:40 | mriedem | and everything runs in the same process | |
| 14:05:36 | mriedem | stephenfin: so yeah i'm not sure how we can run multiple compute services in a functional test but 2 of them share the same numa network config and another doesn't | |
| 14:05:54 | stephenfin | Yup, my point exactly | |
| 14:06:06 | dansmith | we can't | |
| 14:06:28 | dansmith | unless they are things that get reported during service start, | |
| 14:06:43 | dansmith | such that we can set the flag, start the service, have it cache/report a thing, and then change it for the next start | |
| 14:06:53 | dansmith | but that's fragile and very few things fit that pattern | |
| 14:07:20 | dansmith | lol | |
| 14:07:34 | mdbooth | Who can kickban? | |
| 14:07:35 | sean-k-mooney | dansmith: i was going to suggest the same thing. e.g. set conf values for compute node 1 then start then set conf for node 2 then start | |
| 14:07:40 | mriedem | stephenfin: could have a negative test where you start with a host that fits the requested numa affinity and then remove the config, start another host and migrate the server | |
| 14:07:49 | dansmith | sean-k-mooney: that works for very few things | |
| 14:07:56 | mriedem | stephenfin: in that case, what do we expect to happen? the numa filter should kick it out right? | |
| 14:08:26 | sean-k-mooney | dansmith: yep but it might work for stephenfin usecase | |
| 14:08:27 | mriedem | and tests that you've properly plumbed the request spec for the cold migration in conductor | |
| 14:08:35 | stephenfin | mriedem: Yup, that's what I'd expect, anyway | |
| 14:08:50 | bauzas | mriedem: btw. saw your req_spec changes | |
| 14:08:54 | bauzas | mriedem: thanks for this | |
| 14:09:00 | mriedem | and commenting out the ensure* line in nova/conductor/tasks/migrate.py would make the test fail | |
| 14:09:21 | mriedem | bauzas: np, i want all that compat code dead in stein | |
| 14:09:58 | bauzas | mriedem: I'll try to review those quickly | |
| 14:10:01 | mriedem | dansmith: is what i described for a cold migate negative test good enough for you for the move patch? | |
| 14:10:47 | dansmith | mriedem: combined with a positive test where we successfully migrate between two hosts that satisfy the requirement I guess | |
| 14:11:05 | bauzas | btw. what's super important to review for the 3st milestone ? | |
| 14:11:13 | bauzas | for the moment, looking at https://review.openstack.org/#/c/576927/19 | |
| 14:11:14 | stephenfin | mriedem: Wait, actually, it wouldn't complain if I simply removed the config. No config = do nothing special (to preserve backwards compatibility) | |
| 14:11:28 | dansmith | stephenfin: you'll have to change the config I think | |
| 14:11:55 | dansmith | stephenfin: also, probably worth doing the same negative test but where you fill up the other node such that it's not a candidate right? | |
| 14:12:13 | dansmith | like, boot enough things tied to the tunneled numa node that it can't take any more and make sure it properly rejects a new one | |
| 14:12:20 | stephenfin | The latter is probably a better test | |
| 14:12:35 | mriedem | stephenfin: good point, would have to change the config from physnet foo to physnet bar or something on the 2nd host | |
| 14:12:40 | stephenfin | though I'm going to have to rely on the unspecified "things schedule to host numa node 0" behavior for that to work | |
| 14:12:41 | dansmith | I think you probably want both | |
| 14:12:48 | dansmith | right | |
| 14:13:41 | mriedem | if you're going to do multiple tests, i would suggest trying to mix in the various moves, cold migrate, evacuate and unshelve | |
| 14:13:46 | stephenfin | Hmm, if I could attach an interface and try that, it would test so much more. As sean-k-mooney is learning though, there are holes all over the neutron/libvirt fakes | |
| 14:14:27 | mriedem | positive test could be the cold migrate where the hosts have the same config, full fail could be unshelve where the compute host was filled up while the server was shelved offloaded, and negative 2nd host different physnet for evacuate | |
| 14:14:33 | mriedem | that would cover those 3 move ops | |
| 14:14:56 | mriedem | i'm ok with dealing with subsequent port attachments in testing in a follow up | |
| 14:15:15 | mriedem | if you could get those 3 scenarios (1 pos, 2 neg, 3 different move ops) in a patch i think that's pretty solid for now | |
| 14:15:21 | sean-k-mooney | stephenfin: the most annoying thing about the fake dirver is how many thimes we convert back and fort between python objects and xml when using it | |
| 14:15:21 | dansmith | mriedem: we want a positive and negative case of each type of migration, IMHO | |
| 14:15:44 | mriedem | dansmith: ok, if that can get done today... | |
| 14:16:04 | dansmith | mriedem: well, positive and negative of at least one of the same migration ops, I think | |
| 14:16:17 | dansmith | otherwise I think we're cheating | |
| 14:17:27 | mriedem | that should be easy enough with cold migrate as i described above | |
| 14:17:44 | mriedem | pos: start 2 computes with same config to match the request, cold migrate between them | |
| 14:17:55 | dansmith | yup | |
| 14:17:57 | mriedem | neg: start compute 1, boot, change config, start compute 2, migrate, fail | |
| 14:18:05 | dansmith | yup | |
| 14:18:05 | mriedem | stephenfin: ^ | |
| 14:18:55 | stephenfin | mriedem: What would I be changing it to to cause it to fail? | |
| 14:19:02 | dansmith | config | |
| 14:19:17 | stephenfin | *what would I be changing the config to | |
| 14:19:28 | dansmith | change the physnet from foo to bar | |
| 14:19:43 | dansmith | so that there are no nodes reporting physnet foo when you go to migrate | |
| 14:20:02 | stephenfin | That's no an issue though (by design) | |
| 14:20:25 | dansmith | oh right because the op can avoid reporting each physnet | |
| 14:20:30 | dansmith | then you have to do it via fill I guess | |
| 14:20:40 | stephenfin | yup, and to avoid breaking upgrades | |
| 14:22:33 | sean-k-mooney | stephenfin: change the numa mappings | |
| 14:23:10 | dansmith | oh, | |
| 14:23:22 | dansmith | stephenfin: can you report physnet bar on a numa node with no memory to make it fail easy? | |
| 14:23:57 | dansmith | so that when you go to request that physnet, but with one guest node of 1G, you don't fit without having to boot anything on it to fill it? | |
| 14:24:59 | stephenfin | That could work. Lemme try that | |
| 14:25:23 | dansmith | yeah then you could do your config switch trick by just moving the physnet to the stupid node | |
| 14:26:07 | pooja_jadhav | mriedem: h | |
| 14:26:13 | pooja_jadhav | mriedem: hi | |
| 14:27:12 | mriedem | hi | |
| 14:29:07 | sean-k-mooney | stephenfin: i have got to the point now where the failure is now in the assert fails. e.g. rebuild works ... | |
| 14:29:22 | stephenfin | sean-k-mooney: Fancy pushing it up? | |
| 14:30:35 | sean-k-mooney | want me to commit it as a seperate patch and we can quash them later or replace your existing patch | |
| 14:30:55 | stephenfin | sean-k-mooney: Nah, squash and stick in co-authored-by sure | |
| 14:35:26 | openstackgerrit | Merged openstack/nova stable/pike: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580523 | |
| 14:36:34 | openstackgerrit | sean mooney proposed openstack/nova master: WIP: Add functional tests for rebuild with NUMA networks https://review.openstack.org/585385 | |
| 14:42:18 | pooja_jadhav | mriedem: I have done shared storage to test live migrate API, but still its trying shh the dest host.. not getting what action needs to be taken here? | |
| 14:45:14 | pooja_jadhav | mriedem: For logs - http://paste.openstack.org/show/726619/ | |
| 14:45:38 | pooja_jadhav | mriedem: Can u please help me to resolve this issue? | |
| 14:54:05 | mriedem | bauzas: if you're looking for anything to review for FF, my portbinding live migration series has 3 changes left to complete the bp and the bottom patch has a +2 https://review.openstack.org/#/c/551371/ | |
| 14:54:18 | mriedem | needless to say i'd like to see the end of this one https://review.openstack.org/#/q/topic:bp/neutron-new-port-binding-api+(status:open+OR+status:merged) | |
| 14:54:38 | bauzas | mriedem: yeah, okay | |
| 14:54:45 | bauzas | mriedem: thanks | |
| 14:54:57 | bauzas | mriedem: easier to review than the reshape API change :( | |
| 14:57:21 | mriedem | sean-k-mooney: mdbooth: so i'm assuming it's still a known problem if you hard reboot a guest, it's eth config for sriov ports attached could randomly change since we rebuild the guest xml, is that right? similar to how mountpoints for block devices can change on hard reboot, and why we have virtual device tags for that | |
| 14:57:52 | mriedem | reply when you can, but i've got to run out for a few hours | |
| 14:58:10 | sean-k-mooney | mriedem: ya we dont guarentee nic ordering because libvirt does not so it can change on a reboot | |
| 14:59:55 | sean-k-mooney | mriedem_away: if you need to know the ordering then ya you use tags and discover it from metadata. it only really is an issue if you reboot after attaching or detaching an interface. | |
| 15:00:19 | mriedem_away | yup that's what i thought, just making sure | |
| 15:01:02 | stephenfin | If I cold migrate, am I guaranteed to end up on a different host? | |
| 15:01:15 | stephenfin | assuming I don't specify a specific host to cold migrate to (if that's a thing) | |
| 15:01:17 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Extract base functional test case from test_direct https://review.openstack.org/585777 | |
| 15:01:18 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use base test in placement functional tests https://review.openstack.org/585778 | |
| 15:01:23 | sean-k-mooney | stephenfin: i think so | |
| 15:02:24 | sean-k-mooney | stephenfin: i think all migrates will give that guarentee. resize i think could land on the same host but cold migrate should result in an actual migration | |