| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-08 | |||
| 11:50:50 | sean-k-mooney | artom: hi | |
| 11:51:35 | artom | sean-k-mooney, heya - live migration with NUMA (either pinning or actual NUMA topologies) can be tested in the Intel NFV CI, right? | |
| 11:51:42 | sean-k-mooney | sorry i have been on vacation since last tuseday. my mum ha a minor operation so i was being a good son and makeing her tea and mining the house all week | |
| 11:51:52 | artom | sean-k-mooney, awww :) | |
| 11:51:57 | sean-k-mooney | artom: so yes nfv ci | |
| 11:52:11 | artom | Hope she's doing well | |
| 11:53:03 | sean-k-mooney | artom: she is doing fine. am yes you could test numa + livemigration in the intel nfv ci but i think the livemigration tests are not in cluded in teh tempest regex in the multinode job | |
| 11:53:49 | theanalyst | t | |
| 11:54:19 | artom | sean-k-mooney, could you point me to the code? | |
| 11:55:04 | artom | https://github.com/openstack/intel-nfv-ci-tests I assume | |
| 11:55:12 | sean-k-mooney | artom: the ci is currently configured using an internal git repo but im going to take a look at the tempest.conf | |
| 11:55:25 | sean-k-mooney | artom: that is a part of it but not the ci config | |
| 11:55:41 | sean-k-mooney | artom: that host our tempest plugin | |
| 11:56:28 | sean-k-mooney | artom: tempest-dsvm-intel-nfv-xenial/ uses that plugin but its not multinode | |
| 11:56:47 | artom | sean-k-mooney, does it go through gerrit? My plan is to write a live migration NUMA test (unless those already exist? A quick look at the code doesn't turn anything up), and make it Depends-On: my own patches for that | |
| 11:57:49 | sean-k-mooney | artom: the statdard livemigration test would partly test this. the tempest-dsvm-ovsdpdk-nfv-networking-xenial replaces the default flvors with ones that use cpu pinning + hugepages + 2 numa nodes | |
| 11:58:27 | sean-k-mooney | artom: this is one of the tempest.confs from a run on the ci http://52.27.155.124/portland/2018-04-27/553072/6/check/tempest-dsvm-ovsdpdk-nfv-networking-xenial/ff9bb78/logs/tempest_conf.txt.gz | |
| 11:59:00 | sean-k-mooney | so yes live_migration = False is set | |
| 11:59:45 | sean-k-mooney | we could modify that via a patch to networking-ovs-dpdk to enable livemigration via its devstack plugin. | |
| 11:59:51 | artom | sean-k-mooney, so at least as a first step I could need to get live_migration set to True? | |
| 12:00:04 | artom | *would need | |
| 12:00:39 | sean-k-mooney | artom: yep, we could also do that via changing the internal repo but i dont have access to it because i dont know where it is now | |
| 12:01:04 | artom | sean-k-mooney, heh, are the contacts listed on https://wiki.openstack.org/wiki/ThirdPartySystems/Intel_NFV_CI still good? | |
| 12:01:22 | sean-k-mooney | networking-ovs-dpdk however i have full contol over. yes the contacts are correct | |
| 12:02:10 | artom | sean-k-mooney, wait, so which one do you *not* have access to? | |
| 12:03:48 | sean-k-mooney | artom: i do not have acess to the zuul 2.5 configure repo that stores the job config. i am the maintainer of networking-ovs-dpdk and have full aceess to modify the job via that repo using the devstack plugin | |
| 12:04:12 | artom | sean-k-mooney, ah, gotcha | |
| 12:04:20 | artom | And those are both internal Intel, right? | |
| 12:05:01 | sean-k-mooney | networking-ovs-dpdk is hosted by openstack infra but is not technically part of openstack. https://github.com/openstack/networking-ovs-dpdk | |
| 12:05:29 | sean-k-mooney | artom: we should be able to use the test-config section of the devstack plugin to enable live migration https://github.com/openstack-dev/devstack/blob/master/doc/source/plugins.rst#pluginsh-contract | |
| 12:05:39 | artom | Also, http://52.27.155.124/portland/2018-04-20/563037/2/check/tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial/3e525a4/logs/tempest_conf.txt.gz seems to enable live_migration? | |
| 12:06:08 | artom | Which woujld | |
| 12:06:22 | artom | Which wou45ld | |
| 12:06:41 | artom | Which would make sense since it's multinode, and the other one isn't | |
| 12:06:51 | sean-k-mooney | tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial used to be the same job just run against neutron | |
| 12:07:26 | sean-k-mooney | sorry yes i ment tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial | |
| 12:07:29 | artom | (Excuse the random characters, daughter decided my laptop is a toy) | |
| 12:07:54 | sean-k-mooney | artom: no worries | |
| 12:08:09 | sean-k-mooney | still more understandable then my spelling at times | |
| 12:11:29 | sean-k-mooney | artom: do you know if devstack-gate sets any env varibles that we could use to enable "ci testing" config changes within the devstack plugin. | |
| 12:11:57 | artom | sean-k-mooney, I'm not sure I understand the question | |
| 12:13:09 | sean-k-mooney | well if we add the sections to enable livemirgation in the networking-ovs-dpdk devstack plugin i am wondering is there a way to enable it automatically in a ci env and not on a normal install | |
| 12:14:00 | sean-k-mooney | if we are just setting tempest settings then there is no real harm always doing it i was just wondering if there was a way to detect that we are running in a ci | |
| 12:15:11 | mriedem | sean-k-mooney: https://github.com/openstack-infra/devstack-gate/blob/b3da8a393c68cc62924ee2f752f442d5c85ea8ed/devstack-vm-gate.sh#L566 | |
| 12:15:45 | mriedem | all non-grenade multinode jobs run live migration by default | |
| 12:15:55 | mriedem | grenade doesn't because grenade by default only runs smoke tests, and live migration tests aren't smoke tests | |
| 12:16:14 | mriedem | we do have a grenade job specifically that tests just live migration back and forth from n-1 to n and back | |
| 12:17:07 | artom | mriedem, I suppose I could just test numa-aware live migration with cpu pinning since we don't need special hardware for that, but there'd be no way to assert anything about the instance XML, which is kinda what we want | |
| 12:17:28 | sean-k-mooney | mriedem: ah you are right looking at http://52.27.155.124/portland/2018-04-27/553072/6/check/tempest-dsvm-multinode-ovsdpdk-nfv-networking-xenial/943bdb9/logs/tempest_conf.txt.gz livemigration is turned on the quest then is why is it not running those tests | |
| 12:18:08 | mriedem | sean-k-mooney: blacklist? | |
| 12:18:22 | sean-k-mooney | mriedem: proably ill have to look into that | |
| 12:18:47 | mriedem | [[ ^(?!.*(?:tempest\.scenario\.test_network_advanced_server_ops\.TestNetworkAdvancedServerOps\.test_server_connectivity_suspend_resume.*)|(?:tempest\.scenario\.test_network_advanced_server_ops\.TestNetworkAdvancedServerOps\.test_server_connectivity_pause_unpause.*)|(?:.*\.admin\.test_live_migration.*)|(?:.*\.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert.*)|(?:.*\.TestNetworkAdvancedServerOps.test | |
| 12:18:47 | mriedem | ver_connectivity_cold_migration.*)|(?:.*\.TestNetworkAdvancedServerOps.test_server_connectivity_resize.*)|(?:.*\.TestNetworkBasicOps\.test_update_router_admin_state.*)|(?:.*\.TestNetworkBasicOps\.test_network_basic_op.*)|(?:.*\.TestNetworkBasicOps\.test_update_instance_port_admin_state.*))((?:tempest\.scenario\.test_network_basic_ops.*)|(?:tempest\.scenario\.test_network_advanced_server_ops.*)|(?:tempest\.api\.compute\.admin\ | |
| 12:18:47 | mriedem | t_live_migration.*)|(?:tempest\.api\.network.*)).*$ != '' ]] | |
| 12:19:02 | mriedem | yes it's blacklisted live migration | |
| 12:19:08 | sean-k-mooney | artom: you need a multi numa host vm as we have a restiction in the libvirt drivver that maps each guest numa node to a different host numa node | |
| 12:19:12 | artom | Which makes sense, because it's broken | |
| 12:19:37 | sean-k-mooney | artom: actully the reson the multinode job was created was to test livemigration | |
| 12:19:53 | artom | But... we know it doens't work :) | |
| 12:19:58 | mriedem | artom: what do you need to get out of the instance xml? | |
| 12:20:09 | mriedem | the numa config? | |
| 12:20:10 | sean-k-mooney | artom: it was turned off temporaly feburay last year and then the ci moved team and i guess it never got turned back on | |
| 12:20:12 | artom | mriedem, yeah | |
| 12:20:20 | artom | sean-k-mooney, how did it ever pass before? | |
| 12:20:26 | mriedem | artom: we have the instance diagnostics api... | |
| 12:20:41 | sean-k-mooney | artom: livemigration works it just does not respect pinning after | |
| 12:20:53 | artom | sean-k-mooney, ah, indeed | |
| 12:21:06 | sean-k-mooney | e.g. the vm will actully livemigrate we just broke all SLAs | |
| 12:21:36 | mriedem | artom: you could potentially add a numa_details field to the instance diagnostics response and use that for validating things in tempest (likely a tempest plugin) | |
| 12:21:38 | artom | mriedem, I don't think those are enough | |
| 12:22:01 | sean-k-mooney | mriedem: the issue with multi numa testing in the upstream ci is we cannot spawn a 2 numa node guest on a host with 1 numa node | |
| 12:22:10 | sean-k-mooney | using the libvirt driver | |
| 12:22:39 | sean-k-mooney | that is a limitaion that i personally think should never have been there but that is a different issue | |
| 12:23:30 | artom | sean-k-mooney, so is the intel CI multi-physical-node? Or multinode on VMs, so we can't have VMs with more than 1 node? | |
| 12:24:14 | artom | mriedem, would it really be a plugin at that point though? If it's 100% through the API, it's legit tempest | |
| 12:24:27 | sean-k-mooney | artom: mriedem: all the host vms for the intel nfv ci have 2 numa nodes to allow numa testing, the phyical hosts have 2-4 numa nodes dending on the server the nodepool vm lands on | |
| 12:24:45 | mriedem | artom: meaning, it's only compute api | |
| 12:25:01 | mriedem | there isn't a real need to make all other projects gate on a numa test in the common tempest repo | |
| 12:25:39 | artom | mriedem, so how does it work currently for nova-only tempest tests? | |
| 12:26:05 | mriedem | those are legacy | |
| 12:26:13 | mriedem | and some are interop tests | |
| 12:26:21 | mriedem | the real basic stuff is interop | |
| 12:26:21 | artom | mriedem, oh, they're not accepting new tests that are single-service? | |
| 12:26:31 | mriedem | depends | |
| 12:26:46 | artom | Christ, it's getting less useful by the minute | |
| 12:26:54 | artom | (Sorry tempest folks!) | |
| 12:27:03 | mriedem | i just know that over time, single service, non-interop tests were supposed to be moved into the project tree or a tempest plugin for that repo | |
| 12:27:27 | mriedem | most other projects already have tempest plugins, i know cinder and neutron have had their own for a long time | |
| 12:27:50 | artom | I suppose it kinda make sense... | |
| 12:28:11 | artom | Keep the "useful to everyone" stuff in-tree, the rest can be out of scope in plugins | |
| 12:28:16 | artom | Anyways | |
| 12:28:32 | artom | So, I think first step for me is to get live migration re-enabled in the intel NFV CI | |
| 12:28:35 | mriedem | i'm no QA gate keeper, but just don't be surprised if that's what they tell you | |
| 12:29:04 | artom | They'll pass, sortof | |
| 12:29:28 | sean-k-mooney | artom: do you want to test multi numa guests or just guest with a numa topology | |
| 12:29:46 | artom | sean-k-mooney, uh, there's a difference? | |
| 12:29:46 | sean-k-mooney | artom: hw:numa_nodes=1 should work in the upstream ci | |
| 12:29:54 | artom | Ah, in that sense | |
| 12:29:55 | artom | Hrmm | |
| 12:29:59 | artom | True, true | |