| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-26 | |||
| 19:16:38 | artom | dansmith, yeah, but why? | |
| 19:16:58 | artom | This is all just a "forcing" function to make sure that when I live migrate it, the XML will have to get updates | |
| 19:17:02 | dansmith | artom: well, if you put it on the second host, I'll ask if you're sure it would have worked if co-located | |
| 19:17:10 | sean-k-mooney | we are litrally adding a feature this cycle to allow this. | |
| 19:17:14 | dansmith | artom: and if it is co-located, I'll ask if it would have worked if by itself :D | |
| 19:17:29 | dansmith | artom: ah, right, forgot about the migration | |
| 19:17:41 | dansmith | artom: so you an migrate it to and fro and cover both cases right? | |
| 19:17:48 | sean-k-mooney | and its marked as implemented https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | |
| 19:17:53 | dansmith | i.e. migrate twice instead of boot thrice | |
| 19:18:01 | artom | It's about forcing the migration to move the server to a cell that as different CPU numbers | |
| 19:18:29 | sean-k-mooney | you are trying to force the xml to be regenerated | |
| 19:18:33 | artom | Yeah | |
| 19:18:50 | dansmith | artom: right, so two migrations will cover the bases of migrating to an empty host which could take anything and to a host where it can't (as long as it didn't choose the same by luck) | |
| 19:18:56 | artom | The idea is to have a server with 2 NUMA cells: 2 CPUs and 3 CPUs | |
| 19:19:02 | artom | And another with just a single cell: 2 CPUs | |
| 19:19:15 | sean-k-mooney | so why not use the host/hypervioer_host name parmaters on server create | |
| 19:19:30 | artom | But such that the 2-CPU cells have different physical CPUs in them | |
| 19:19:38 | sean-k-mooney | it will allow you slect the host and it will use the schdler to validate it is sutable | |
| 19:19:46 | artom | So host2: [0, 1], host3: [0, 1, 2] [3, 4] | |
| 19:20:14 | artom | "Fill" the 3-CPU cell with a first instance | |
| 19:20:20 | artom | Then boot the 2-CPU instance | |
| 19:20:29 | dansmith | artom: is this for a tempest test or a functional test? | |
| 19:20:35 | artom | I don't care where it goes, as long as I live migrate it, its CPU pins have changed | |
| 19:20:38 | artom | dansmith, functional | |
| 19:21:16 | dansmith | okay, I think we have other tests that migrate twice just to make sure we hit both hosts, regardless of the ordering | |
| 19:21:27 | dansmith | which I prefer to modifying the scheduler, if it will cover us | |
| 19:21:45 | dansmith | targeting the build works too I guess, but it's also not the more realistic case you're trying to replicate | |
| 19:21:54 | sean-k-mooney | we currently dont have any live migration test the suceed and use the fakelibvirt dirver in teh functional tests | |
| 19:22:08 | artom | Yey, I'm first :( | |
| 19:22:32 | sean-k-mooney | so ignrogn the lack of live migration tests i dont see what the big deal is | |
| 19:22:56 | sean-k-mooney | you stare each compute node with a different config with non overlaping cpus | |
| 19:23:03 | sean-k-mooney | in the dedicate_cpu_set | |
| 19:23:10 | sean-k-mooney | then spawn a singel vm | |
| 19:23:17 | sean-k-mooney | and migrated to the ohter node | |
| 19:23:23 | sean-k-mooney | and your done | |
| 19:23:39 | artom | Uhh, literally *ALL* tests are failing with "keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL" | |
| 19:23:42 | artom | I think devstack bork | |
| 19:23:50 | sean-k-mooney | you need to rebase | |
| 19:23:59 | sean-k-mooney | this is fixed on master | |
| 19:24:38 | artom | ack | |
| 19:24:45 | artom | When I push the new func test | |
| 19:24:55 | sean-k-mooney | are you goint to do ^ | |
| 19:24:56 | artom | I gotta run for a daughter school thing (yes, she's starting shool!) | |
| 19:25:10 | artom | sean-k-mooney, can't have different configs in func tests | |
| 19:25:19 | sean-k-mooney | yes you can | |
| 19:25:32 | artom | Nope, CONF is global | |
| 19:25:43 | sean-k-mooney | you need to chente the config flag and start the compute agent serially | |
| 19:26:11 | sean-k-mooney | it does not have to be with mocks and we read most of the values on start up | |
| 19:26:20 | dansmith | sean-k-mooney: that only works for a subset of cases | |
| 19:30:17 | sean-k-mooney | dansmith: yes but i think it will work in this case. i do not belview we parse the config on ever iteration of the perodic tasks. | |
| 19:31:03 | sean-k-mooney | if we do we should stop and cache it. we dont support change this config at runtime. | |
| 19:35:28 | dansmith | it's not just the periodic that matters of course | |
| 19:36:10 | dansmith | % fgrep 'CONF.' -r nova/virt/hardware.py | wc -l | |
| 19:36:11 | dansmith | 9 | |
| 19:36:12 | sean-k-mooney | true but i think it would work in this case. the simple way would to test this would be just try it | |
| 19:37:46 | sean-k-mooney | at least 7 of those shoudl one be check at star up i think | |
| 19:38:19 | dansmith | I'm just saying, this stuff is spread everywhere | |
| 19:38:22 | sean-k-mooney | i think all of them are just used in init | |
| 19:38:27 | sean-k-mooney | ya it is | |
| 19:38:45 | sean-k-mooney | i just think/hope this is simpler to do then it first seams | |
| 19:39:23 | sean-k-mooney | if we test this simple with conf setting that would be better then create a complicate test setup | |
| 19:44:40 | sean-k-mooney | anyway i replied in https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@304 not sure if that is what ye wanted. | |
| 19:45:06 | sean-k-mooney | ill redeploy with artoms code shortly and start testing it again. | |
| 20:02:36 | TheJulia | mriedem: remember that seemingly weird rebalance race on stable/stein that we spotted with ironic last week? I just spotted it on master branch testing https://d01b2e57f0a56cb7edf0-b6bc206936c08bb07a5f77cfa916a2d4.ssl.cf5.rackcdn.com/678298/4/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/92c65ac/compute1/logs/screen-n-cpu.txt.gz <-- nova 19.1.0 :\ | |
| 20:04:04 | sean-k-mooney | TheJulia: implying that backport the other fix may not resolve the issue | |
| 20:04:23 | sean-k-mooney | assuming on master it still had the db issue | |
| 20:05:23 | TheJulia | indeed :( | |
| 20:06:09 | sean-k-mooney | those errors seam to be indicating that the RP is not found | |
| 20:06:25 | sean-k-mooney | im not sing the db conclit on the compute node uuid | |
| 20:06:49 | TheJulia | seeing? | |
| 20:07:28 | sean-k-mooney | yes :) | |
| 20:07:55 | sean-k-mooney | i dropped the ee it seams | |
| 20:08:22 | TheJulia | no worries, my brain does similar things | |
| 20:08:39 | TheJulia | so without the conflict.... hmmmm | |
| 20:09:50 | sean-k-mooney | but it might be that the other compute node is not recreating the rp | |
| 20:10:07 | sean-k-mooney | or it is but we race | |
| 20:11:58 | TheJulia | I'll go with it is but we race | |
| 20:15:31 | mriedem | umm yeah doesn't look like the same issue, no db conflicts on restart of the service that i see | |
| 20:15:47 | mriedem | i see this: | |
| 20:15:47 | mriedem | Aug 26 18:41:38.367990 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: INFO nova.compute.resource_tracker [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] ComputeNode eabf1567-cb9f-4a93-b77e-44a80ddd50b0 moving from ubuntu-bionic-rax-ord-0010443317 to ubuntu-bionic-rax-ord-0010443319 | |
| 20:16:10 | mriedem | Aug 26 18:41:38.818412 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: INFO nova.compute.resource_tracker [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] ComputeNode 61dbc9c7-828b-4c42-b19c-a3716037965f moving from ubuntu-bionic-rax-ord-0010443317 to ubuntu-bionic-rax-ord-0010443319 | |
| 20:16:27 | mriedem | and then a bunch of errors about resource provider not found | |
| 20:17:08 | mriedem | Aug 26 18:41:38.992017 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: ERROR nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] [req-208b7275-abc4-4445-9551-8edbce49e0e1] Failed to retrieve traits from placement API for resource provider with UUID 61dbc9c7-828b-4c42-b19c-a3716037965f. Got 404: {"errors": [{"status": 404, "request_id": "req-208b7275-abc4-4445-9551-8edbce49e0e1", "detail" | |
| 20:17:08 | mriedem | he resource could not be found.\n\n No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found: No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found ", "title": "Not Found"}]}. | |
| 20:17:42 | sean-k-mooney | it looks like the perodic taks the cleans up allocation ran before the compute node rp was recreated by the other compute service | |
| 20:18:21 | mriedem | the traceback hits get_provider_tree_and_ensure_root which should ensure the provider exists, so i'm not sure why it wouldn't, | |
| 20:18:30 | mriedem | unless it's only looking at it's local provider tree cache and that is stale or something | |
| 20:23:06 | TheJulia | that does seem to possibly be the case. just glancing at records for eabf1567-cb9f-4a93-b77e-44a80ddd50b0, it is all over both compute logs | |
| 20:24:36 | TheJulia | in the same time window it looks like... | |
| 20:29:35 | openstackgerrit | Merged openstack/nova master: Remove descriptions of nonexistent hacking rules https://review.opendev.org/678462 | |
| 20:36:44 | TheJulia | hmmmmm.... I wonder | |
| 20:41:57 | openstackgerrit | Merged openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 20:47:18 | TheJulia | I'm starting to wonder, at least digging at it, if things did't moderately correct themselves at least resource tracker wise as time went on. Looks like everything changed to no longer reserved around 18:46, placement reports still one node reserved 3 minutes later | |
| 21:09:07 | mriedem | i think this is bogus | |
| 21:09:08 | mriedem | Aug 26 18:41:38.992017 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: ERROR nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] [req-208b7275-abc4-4445-9551-8edbce49e0e1] Failed to retrieve traits from placement API for resource provider with UUID 61dbc9c7-828b-4c42-b19c-a3716037965f. Got 404: {"errors": [{"status": 404, "request_id": "req-208b7275-abc4-4445-9551-8edbce49e0e1", "detail" | |
| 21:09:08 | mriedem | he resource could not be found.\n\n No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found: No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found ", "title": "Not Found"}]}. | |
| 21:09:16 | mriedem | because right before that we get inventories and aggregates for the provider | |
| 21:09:28 | mriedem | Aug 26 18:41:38.881026 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: DEBUG nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] Refreshing inventories for resource provider 61dbc9c7-828b-4c42-b19c-a3716037965f {{(pid=747) _refresh_associations /opt/stack/nova/nova/scheduler/client/report.py:761}} | |
| 21:09:35 | mriedem | Aug 26 18:41:38.953685 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: DEBUG nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] Refreshing aggregate associations for resource provider 61dbc9c7-828b-4c42-b19c-a3716037965f, aggregates: None {{(pid=747) _refresh_associations /opt/stack/nova/nova/scheduler/client/report.py:770}} | |
| 21:12:03 | mriedem | unless at the same time, the other host is deleting the resource provider | |