| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-26 | |||
| 19:13:33 | sean-k-mooney | if you wan to force it to the other host i thike we added --host option recently | |
| 19:13:56 | dansmith | artom: I would rather you write the test to be more specific than that if possible | |
| 19:14:13 | artom | dansmith, so force it with a weigher to a host? | |
| 19:14:29 | artom | It doesn't change the mechanics of the test, just the numbers that we assert. | |
| 19:14:31 | sean-k-mooney | is the scenarion that the ifrst instace can oly fit on the first host and the second can fit on either(but not after the first is booted) | |
| 19:14:51 | artom | sean-k-mooney, no, the second can fit on either regardless of the first instance. | |
| 19:14:58 | sean-k-mooney | oh | |
| 19:15:00 | dansmith | artom: yeah, I don't really like altering the behavior in a functional test, but scheduling determinism bugs in functional tests are annoying, | |
| 19:15:15 | dansmith | artom: especially if they work now, but start to fail the obscure assumptions made in a year | |
| 19:15:18 | sean-k-mooney | then use https://github.com/openstack/nova-specs/blob/master/specs/train/approved/add-host-and-hypervisor-hostname-flag-to-create-server.rst | |
| 19:15:27 | artom | dansmith, well, what I currently have is just an if | |
| 19:15:40 | artom | if hostA; assert hostA things; else assert hostB things | |
| 19:15:47 | artom | (really just the number of the pinned CPUs) | |
| 19:15:51 | dansmith | artom: meaning the test tolerates it being either place? | |
| 19:15:53 | sean-k-mooney | i think we merged the code for that a few weeks ago right | |
| 19:15:56 | artom | dansmith, exactly | |
| 19:16:08 | dansmith | artom: kinda seems like that's begging for testing both scenarios, no? | |
| 19:16:26 | dansmith | artom: can you boot a third like the second that will have to go in the spot not taken by the second instance and then assert it al? | |
| 19:16:38 | artom | dansmith, yeah, but why? | |
| 19:16:58 | artom | This is all just a "forcing" function to make sure that when I live migrate it, the XML will have to get updates | |
| 19:17:02 | dansmith | artom: well, if you put it on the second host, I'll ask if you're sure it would have worked if co-located | |
| 19:17:10 | sean-k-mooney | we are litrally adding a feature this cycle to allow this. | |
| 19:17:14 | dansmith | artom: and if it is co-located, I'll ask if it would have worked if by itself :D | |
| 19:17:29 | dansmith | artom: ah, right, forgot about the migration | |
| 19:17:41 | dansmith | artom: so you an migrate it to and fro and cover both cases right? | |
| 19:17:48 | sean-k-mooney | and its marked as implemented https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | |
| 19:17:53 | dansmith | i.e. migrate twice instead of boot thrice | |
| 19:18:01 | artom | It's about forcing the migration to move the server to a cell that as different CPU numbers | |
| 19:18:29 | sean-k-mooney | you are trying to force the xml to be regenerated | |
| 19:18:33 | artom | Yeah | |
| 19:18:50 | dansmith | artom: right, so two migrations will cover the bases of migrating to an empty host which could take anything and to a host where it can't (as long as it didn't choose the same by luck) | |
| 19:18:56 | artom | The idea is to have a server with 2 NUMA cells: 2 CPUs and 3 CPUs | |
| 19:19:02 | artom | And another with just a single cell: 2 CPUs | |
| 19:19:15 | sean-k-mooney | so why not use the host/hypervioer_host name parmaters on server create | |
| 19:19:30 | artom | But such that the 2-CPU cells have different physical CPUs in them | |
| 19:19:38 | sean-k-mooney | it will allow you slect the host and it will use the schdler to validate it is sutable | |
| 19:19:46 | artom | So host2: [0, 1], host3: [0, 1, 2] [3, 4] | |
| 19:20:14 | artom | "Fill" the 3-CPU cell with a first instance | |
| 19:20:20 | artom | Then boot the 2-CPU instance | |
| 19:20:29 | dansmith | artom: is this for a tempest test or a functional test? | |
| 19:20:35 | artom | I don't care where it goes, as long as I live migrate it, its CPU pins have changed | |
| 19:20:38 | artom | dansmith, functional | |
| 19:21:16 | dansmith | okay, I think we have other tests that migrate twice just to make sure we hit both hosts, regardless of the ordering | |
| 19:21:27 | dansmith | which I prefer to modifying the scheduler, if it will cover us | |
| 19:21:45 | dansmith | targeting the build works too I guess, but it's also not the more realistic case you're trying to replicate | |
| 19:21:54 | sean-k-mooney | we currently dont have any live migration test the suceed and use the fakelibvirt dirver in teh functional tests | |
| 19:22:08 | artom | Yey, I'm first :( | |
| 19:22:32 | sean-k-mooney | so ignrogn the lack of live migration tests i dont see what the big deal is | |
| 19:22:56 | sean-k-mooney | you stare each compute node with a different config with non overlaping cpus | |
| 19:23:03 | sean-k-mooney | in the dedicate_cpu_set | |
| 19:23:10 | sean-k-mooney | then spawn a singel vm | |
| 19:23:17 | sean-k-mooney | and migrated to the ohter node | |
| 19:23:23 | sean-k-mooney | and your done | |
| 19:23:39 | artom | Uhh, literally *ALL* tests are failing with "keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL" | |
| 19:23:42 | artom | I think devstack bork | |
| 19:23:50 | sean-k-mooney | you need to rebase | |
| 19:23:59 | sean-k-mooney | this is fixed on master | |
| 19:24:38 | artom | ack | |
| 19:24:45 | artom | When I push the new func test | |
| 19:24:55 | sean-k-mooney | are you goint to do ^ | |
| 19:24:56 | artom | I gotta run for a daughter school thing (yes, she's starting shool!) | |
| 19:25:10 | artom | sean-k-mooney, can't have different configs in func tests | |
| 19:25:19 | sean-k-mooney | yes you can | |
| 19:25:32 | artom | Nope, CONF is global | |
| 19:25:43 | sean-k-mooney | you need to chente the config flag and start the compute agent serially | |
| 19:26:11 | sean-k-mooney | it does not have to be with mocks and we read most of the values on start up | |
| 19:26:20 | dansmith | sean-k-mooney: that only works for a subset of cases | |
| 19:30:17 | sean-k-mooney | dansmith: yes but i think it will work in this case. i do not belview we parse the config on ever iteration of the perodic tasks. | |
| 19:31:03 | sean-k-mooney | if we do we should stop and cache it. we dont support change this config at runtime. | |
| 19:35:28 | dansmith | it's not just the periodic that matters of course | |
| 19:36:10 | dansmith | % fgrep 'CONF.' -r nova/virt/hardware.py | wc -l | |
| 19:36:11 | dansmith | 9 | |
| 19:36:12 | sean-k-mooney | true but i think it would work in this case. the simple way would to test this would be just try it | |
| 19:37:46 | sean-k-mooney | at least 7 of those shoudl one be check at star up i think | |
| 19:38:19 | dansmith | I'm just saying, this stuff is spread everywhere | |
| 19:38:22 | sean-k-mooney | i think all of them are just used in init | |
| 19:38:27 | sean-k-mooney | ya it is | |
| 19:38:45 | sean-k-mooney | i just think/hope this is simpler to do then it first seams | |
| 19:39:23 | sean-k-mooney | if we test this simple with conf setting that would be better then create a complicate test setup | |
| 19:44:40 | sean-k-mooney | anyway i replied in https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@304 not sure if that is what ye wanted. | |
| 19:45:06 | sean-k-mooney | ill redeploy with artoms code shortly and start testing it again. | |
| 20:02:36 | TheJulia | mriedem: remember that seemingly weird rebalance race on stable/stein that we spotted with ironic last week? I just spotted it on master branch testing https://d01b2e57f0a56cb7edf0-b6bc206936c08bb07a5f77cfa916a2d4.ssl.cf5.rackcdn.com/678298/4/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/92c65ac/compute1/logs/screen-n-cpu.txt.gz <-- nova 19.1.0 :\ | |
| 20:04:04 | sean-k-mooney | TheJulia: implying that backport the other fix may not resolve the issue | |
| 20:04:23 | sean-k-mooney | assuming on master it still had the db issue | |
| 20:05:23 | TheJulia | indeed :( | |
| 20:06:09 | sean-k-mooney | those errors seam to be indicating that the RP is not found | |
| 20:06:25 | sean-k-mooney | im not sing the db conclit on the compute node uuid | |
| 20:06:49 | TheJulia | seeing? | |
| 20:07:28 | sean-k-mooney | yes :) | |
| 20:07:55 | sean-k-mooney | i dropped the ee it seams | |
| 20:08:22 | TheJulia | no worries, my brain does similar things | |
| 20:08:39 | TheJulia | so without the conflict.... hmmmm | |
| 20:09:50 | sean-k-mooney | but it might be that the other compute node is not recreating the rp | |
| 20:10:07 | sean-k-mooney | or it is but we race | |
| 20:11:58 | TheJulia | I'll go with it is but we race | |
| 20:15:31 | mriedem | umm yeah doesn't look like the same issue, no db conflicts on restart of the service that i see | |
| 20:15:47 | mriedem | Aug 26 18:41:38.367990 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: INFO nova.compute.resource_tracker [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] ComputeNode eabf1567-cb9f-4a93-b77e-44a80ddd50b0 moving from ubuntu-bionic-rax-ord-0010443317 to ubuntu-bionic-rax-ord-0010443319 | |
| 20:15:47 | mriedem | i see this: | |
| 20:16:10 | mriedem | Aug 26 18:41:38.818412 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: INFO nova.compute.resource_tracker [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] ComputeNode 61dbc9c7-828b-4c42-b19c-a3716037965f moving from ubuntu-bionic-rax-ord-0010443317 to ubuntu-bionic-rax-ord-0010443319 | |
| 20:16:27 | mriedem | and then a bunch of errors about resource provider not found | |