| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-26 | |||
| 18:44:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | |
| 18:44:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 18:44:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 18:44:20 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | |
| 18:44:35 | dansmith | efried: well, I don't know without looking, but let me tell you why it's not built in: | |
| 18:45:17 | dansmith | efried: since OVOs can have only part of the database state loaded, we don't provide a default equality operator because determining equality with missing things may be doable in some cases and not others | |
| 18:45:32 | dansmith | efried: and with a nested object the parent and child may have differing rules there | |
| 18:46:11 | efried | dansmith: ack, thanks for the explanation. | |
| 18:46:25 | mriedem | true story, i was writing some ovo compare stuff for cross cell resize and had masked bugs because something was passing b/c one of the objects didn't have a field loaded | |
| 18:46:40 | dansmith | efried: so for an object with primitive field types, then yes that should work, but if it has any nested objects, it won't | |
| 18:46:59 | openstackgerrit | Merged openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | |
| 18:47:42 | dansmith | efried: for some objects, "does the row id match" is good enough, and for others, you need to compare the contents in more depth | |
| 18:53:25 | efried | dansmith: so for this Resource thing, we're going to be asking the virt driver to populate the (local) resources associated with a provider every time RT update runs (so on periodics, instance ops, etc). I would think we want to avoid writing the same records back to the database every iteration. | |
| 18:53:40 | efried | Which means we need to be able to compare with existing | |
| 18:53:56 | efried | what decides how much of the object gets loaded from the db? | |
| 18:54:39 | dansmith | efried: the code that loads it from the db | |
| 18:54:53 | dansmith | efried: this is a performance-hating blob-based object right? in that case, it always comes out whole | |
| 18:55:20 | efried | The touchpoint with the db is that InstanceExtra gets a resources field which is of type ResourceList | |
| 18:56:17 | efried | so ultimately I guess we need to be able to compare a ResourceList we get/build from the virt driver with whatever's in Instance.extras | |
| 19:00:57 | sean-k-mooney | dansmith: yes although i was not sure if we already covered it on irc | |
| 19:01:06 | sean-k-mooney | ill resond on the revew https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py | |
| 19:11:29 | artom | In an artificial func test scenario with 2 compute hosts, we boot 2 instances. The first one can only fit on one host, the second one on either. Can we be 100% it will be placed on the host that does *not* have the first instance? | |
| 19:11:40 | artom | 100% sure? | |
| 19:11:53 | artom | IOW, in that contrived scenario, how deterministic is scheduling? | |
| 19:12:10 | dansmith | if it can't fit, it should be fully deterministic | |
| 19:12:34 | artom | dansmith, the first instance, yeah. But the second instance can go on either host. | |
| 19:12:36 | dansmith | there are a couple tests that add a weigher which always prefers the first host to help it be deterministic in the order it selects even the first one | |
| 19:12:57 | dansmith | artom: so you're asking if the second one, which will *still* fit will go to one specifically? | |
| 19:13:17 | sean-k-mooney | if you boot it in that order tehn its determistic | |
| 19:13:23 | artom | dansmith, yeah. The scheduler's choice is "host A with another instance" or "host B with no instances" | |
| 19:13:28 | dansmith | sean-k-mooney: not if host_subset_size != 1 | |
| 19:13:33 | sean-k-mooney | if you wan to force it to the other host i thike we added --host option recently | |
| 19:13:56 | dansmith | artom: I would rather you write the test to be more specific than that if possible | |
| 19:14:13 | artom | dansmith, so force it with a weigher to a host? | |
| 19:14:29 | artom | It doesn't change the mechanics of the test, just the numbers that we assert. | |
| 19:14:31 | sean-k-mooney | is the scenarion that the ifrst instace can oly fit on the first host and the second can fit on either(but not after the first is booted) | |
| 19:14:51 | artom | sean-k-mooney, no, the second can fit on either regardless of the first instance. | |
| 19:14:58 | sean-k-mooney | oh | |
| 19:15:00 | dansmith | artom: yeah, I don't really like altering the behavior in a functional test, but scheduling determinism bugs in functional tests are annoying, | |
| 19:15:15 | dansmith | artom: especially if they work now, but start to fail the obscure assumptions made in a year | |
| 19:15:18 | sean-k-mooney | then use https://github.com/openstack/nova-specs/blob/master/specs/train/approved/add-host-and-hypervisor-hostname-flag-to-create-server.rst | |
| 19:15:27 | artom | dansmith, well, what I currently have is just an if | |
| 19:15:40 | artom | if hostA; assert hostA things; else assert hostB things | |
| 19:15:47 | artom | (really just the number of the pinned CPUs) | |
| 19:15:51 | dansmith | artom: meaning the test tolerates it being either place? | |
| 19:15:53 | sean-k-mooney | i think we merged the code for that a few weeks ago right | |
| 19:15:56 | artom | dansmith, exactly | |
| 19:16:08 | dansmith | artom: kinda seems like that's begging for testing both scenarios, no? | |
| 19:16:26 | dansmith | artom: can you boot a third like the second that will have to go in the spot not taken by the second instance and then assert it al? | |
| 19:16:38 | artom | dansmith, yeah, but why? | |
| 19:16:58 | artom | This is all just a "forcing" function to make sure that when I live migrate it, the XML will have to get updates | |
| 19:17:02 | dansmith | artom: well, if you put it on the second host, I'll ask if you're sure it would have worked if co-located | |
| 19:17:10 | sean-k-mooney | we are litrally adding a feature this cycle to allow this. | |
| 19:17:14 | dansmith | artom: and if it is co-located, I'll ask if it would have worked if by itself :D | |
| 19:17:29 | dansmith | artom: ah, right, forgot about the migration | |
| 19:17:41 | dansmith | artom: so you an migrate it to and fro and cover both cases right? | |
| 19:17:48 | sean-k-mooney | and its marked as implemented https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | |
| 19:17:53 | dansmith | i.e. migrate twice instead of boot thrice | |
| 19:18:01 | artom | It's about forcing the migration to move the server to a cell that as different CPU numbers | |
| 19:18:29 | sean-k-mooney | you are trying to force the xml to be regenerated | |
| 19:18:33 | artom | Yeah | |
| 19:18:50 | dansmith | artom: right, so two migrations will cover the bases of migrating to an empty host which could take anything and to a host where it can't (as long as it didn't choose the same by luck) | |
| 19:18:56 | artom | The idea is to have a server with 2 NUMA cells: 2 CPUs and 3 CPUs | |
| 19:19:02 | artom | And another with just a single cell: 2 CPUs | |
| 19:19:15 | sean-k-mooney | so why not use the host/hypervioer_host name parmaters on server create | |
| 19:19:30 | artom | But such that the 2-CPU cells have different physical CPUs in them | |
| 19:19:38 | sean-k-mooney | it will allow you slect the host and it will use the schdler to validate it is sutable | |
| 19:19:46 | artom | So host2: [0, 1], host3: [0, 1, 2] [3, 4] | |
| 19:20:14 | artom | "Fill" the 3-CPU cell with a first instance | |
| 19:20:20 | artom | Then boot the 2-CPU instance | |
| 19:20:29 | dansmith | artom: is this for a tempest test or a functional test? | |
| 19:20:35 | artom | I don't care where it goes, as long as I live migrate it, its CPU pins have changed | |
| 19:20:38 | artom | dansmith, functional | |
| 19:21:16 | dansmith | okay, I think we have other tests that migrate twice just to make sure we hit both hosts, regardless of the ordering | |
| 19:21:27 | dansmith | which I prefer to modifying the scheduler, if it will cover us | |
| 19:21:45 | dansmith | targeting the build works too I guess, but it's also not the more realistic case you're trying to replicate | |
| 19:21:54 | sean-k-mooney | we currently dont have any live migration test the suceed and use the fakelibvirt dirver in teh functional tests | |
| 19:22:08 | artom | Yey, I'm first :( | |
| 19:22:32 | sean-k-mooney | so ignrogn the lack of live migration tests i dont see what the big deal is | |
| 19:22:56 | sean-k-mooney | you stare each compute node with a different config with non overlaping cpus | |
| 19:23:03 | sean-k-mooney | in the dedicate_cpu_set | |
| 19:23:10 | sean-k-mooney | then spawn a singel vm | |
| 19:23:17 | sean-k-mooney | and migrated to the ohter node | |
| 19:23:23 | sean-k-mooney | and your done | |
| 19:23:39 | artom | Uhh, literally *ALL* tests are failing with "keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL" | |
| 19:23:42 | artom | I think devstack bork | |
| 19:23:50 | sean-k-mooney | you need to rebase | |
| 19:23:59 | sean-k-mooney | this is fixed on master | |
| 19:24:38 | artom | ack | |
| 19:24:45 | artom | When I push the new func test | |
| 19:24:55 | sean-k-mooney | are you goint to do ^ | |
| 19:24:56 | artom | I gotta run for a daughter school thing (yes, she's starting shool!) | |
| 19:25:10 | artom | sean-k-mooney, can't have different configs in func tests | |
| 19:25:19 | sean-k-mooney | yes you can | |
| 19:25:32 | artom | Nope, CONF is global | |
| 19:25:43 | sean-k-mooney | you need to chente the config flag and start the compute agent serially | |
| 19:26:11 | sean-k-mooney | it does not have to be with mocks and we read most of the values on start up | |
| 19:26:20 | dansmith | sean-k-mooney: that only works for a subset of cases | |
| 19:30:17 | sean-k-mooney | dansmith: yes but i think it will work in this case. i do not belview we parse the config on ever iteration of the perodic tasks. | |
| 19:31:03 | sean-k-mooney | if we do we should stop and cache it. we dont support change this config at runtime. | |