| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-26 | |||
| 17:40:03 | sean-k-mooney | robustify-evacuate you say. how did that go | |
| 17:43:32 | dansmith | sean-k-mooney: you should have seen it before robustification | |
| 17:45:12 | sean-k-mooney | i unfortunetly did see some of it. but i was only a year or so into dealing with openstack at that point and was still mainly looking at neturon at that point | |
| 17:45:58 | sean-k-mooney | we used to pass a log more dicts of random garbage around back then | |
| 17:46:21 | mriedem | and lo before the times of the great robustification there was much deleting of servers and gnashing of teeth | |
| 17:48:17 | sean-k-mooney | a lot of the legacy systems in nova are kind of like rabbitmq. sure they can cause you pain but they generally work well enough that its more pain to replace them with something else so we dont. better the dragons you know and all that | |
| 18:32:27 | artom | *sigh* | |
| 18:32:31 | artom | new_ != _new | |
| 18:39:31 | efried | artom: neither one is reserved or anything, right? | |
| 18:40:10 | artom | efried, hah, no, this isn't Python, it's migration context prefix grumbling | |
| 18:42:56 | dansmith | sean-k-mooney: you know we're looking for your input here right? https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py | |
| 18:44:12 | efried | dansmith: In order to compare two OVOs, it is (apparently) necessary to overrid __eq__. Is nova.objects.numa.all_things_equal the accepted way to do this? | |
| 18:44:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 18:44:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 18:44:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | |
| 18:44:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | |
| 18:44:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 18:44:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 18:44:20 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | |
| 18:44:35 | dansmith | efried: well, I don't know without looking, but let me tell you why it's not built in: | |
| 18:45:17 | dansmith | efried: since OVOs can have only part of the database state loaded, we don't provide a default equality operator because determining equality with missing things may be doable in some cases and not others | |
| 18:45:32 | dansmith | efried: and with a nested object the parent and child may have differing rules there | |
| 18:46:11 | efried | dansmith: ack, thanks for the explanation. | |
| 18:46:25 | mriedem | true story, i was writing some ovo compare stuff for cross cell resize and had masked bugs because something was passing b/c one of the objects didn't have a field loaded | |
| 18:46:40 | dansmith | efried: so for an object with primitive field types, then yes that should work, but if it has any nested objects, it won't | |
| 18:46:59 | openstackgerrit | Merged openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | |
| 18:47:42 | dansmith | efried: for some objects, "does the row id match" is good enough, and for others, you need to compare the contents in more depth | |
| 18:53:25 | efried | dansmith: so for this Resource thing, we're going to be asking the virt driver to populate the (local) resources associated with a provider every time RT update runs (so on periodics, instance ops, etc). I would think we want to avoid writing the same records back to the database every iteration. | |
| 18:53:40 | efried | Which means we need to be able to compare with existing | |
| 18:53:56 | efried | what decides how much of the object gets loaded from the db? | |
| 18:54:39 | dansmith | efried: the code that loads it from the db | |
| 18:54:53 | dansmith | efried: this is a performance-hating blob-based object right? in that case, it always comes out whole | |
| 18:55:20 | efried | The touchpoint with the db is that InstanceExtra gets a resources field which is of type ResourceList | |
| 18:56:17 | efried | so ultimately I guess we need to be able to compare a ResourceList we get/build from the virt driver with whatever's in Instance.extras | |
| 19:00:57 | sean-k-mooney | dansmith: yes although i was not sure if we already covered it on irc | |
| 19:01:06 | sean-k-mooney | ill resond on the revew https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py | |
| 19:11:29 | artom | In an artificial func test scenario with 2 compute hosts, we boot 2 instances. The first one can only fit on one host, the second one on either. Can we be 100% it will be placed on the host that does *not* have the first instance? | |
| 19:11:40 | artom | 100% sure? | |
| 19:11:53 | artom | IOW, in that contrived scenario, how deterministic is scheduling? | |
| 19:12:10 | dansmith | if it can't fit, it should be fully deterministic | |
| 19:12:34 | artom | dansmith, the first instance, yeah. But the second instance can go on either host. | |
| 19:12:36 | dansmith | there are a couple tests that add a weigher which always prefers the first host to help it be deterministic in the order it selects even the first one | |
| 19:12:57 | dansmith | artom: so you're asking if the second one, which will *still* fit will go to one specifically? | |
| 19:13:17 | sean-k-mooney | if you boot it in that order tehn its determistic | |
| 19:13:23 | artom | dansmith, yeah. The scheduler's choice is "host A with another instance" or "host B with no instances" | |
| 19:13:28 | dansmith | sean-k-mooney: not if host_subset_size != 1 | |
| 19:13:33 | sean-k-mooney | if you wan to force it to the other host i thike we added --host option recently | |
| 19:13:56 | dansmith | artom: I would rather you write the test to be more specific than that if possible | |
| 19:14:13 | artom | dansmith, so force it with a weigher to a host? | |
| 19:14:29 | artom | It doesn't change the mechanics of the test, just the numbers that we assert. | |
| 19:14:31 | sean-k-mooney | is the scenarion that the ifrst instace can oly fit on the first host and the second can fit on either(but not after the first is booted) | |
| 19:14:51 | artom | sean-k-mooney, no, the second can fit on either regardless of the first instance. | |
| 19:14:58 | sean-k-mooney | oh | |
| 19:15:00 | dansmith | artom: yeah, I don't really like altering the behavior in a functional test, but scheduling determinism bugs in functional tests are annoying, | |
| 19:15:15 | dansmith | artom: especially if they work now, but start to fail the obscure assumptions made in a year | |
| 19:15:18 | sean-k-mooney | then use https://github.com/openstack/nova-specs/blob/master/specs/train/approved/add-host-and-hypervisor-hostname-flag-to-create-server.rst | |
| 19:15:27 | artom | dansmith, well, what I currently have is just an if | |
| 19:15:40 | artom | if hostA; assert hostA things; else assert hostB things | |
| 19:15:47 | artom | (really just the number of the pinned CPUs) | |
| 19:15:51 | dansmith | artom: meaning the test tolerates it being either place? | |
| 19:15:53 | sean-k-mooney | i think we merged the code for that a few weeks ago right | |
| 19:15:56 | artom | dansmith, exactly | |
| 19:16:08 | dansmith | artom: kinda seems like that's begging for testing both scenarios, no? | |
| 19:16:26 | dansmith | artom: can you boot a third like the second that will have to go in the spot not taken by the second instance and then assert it al? | |
| 19:16:38 | artom | dansmith, yeah, but why? | |
| 19:16:58 | artom | This is all just a "forcing" function to make sure that when I live migrate it, the XML will have to get updates | |
| 19:17:02 | dansmith | artom: well, if you put it on the second host, I'll ask if you're sure it would have worked if co-located | |
| 19:17:10 | sean-k-mooney | we are litrally adding a feature this cycle to allow this. | |
| 19:17:14 | dansmith | artom: and if it is co-located, I'll ask if it would have worked if by itself :D | |
| 19:17:29 | dansmith | artom: ah, right, forgot about the migration | |
| 19:17:41 | dansmith | artom: so you an migrate it to and fro and cover both cases right? | |
| 19:17:48 | sean-k-mooney | and its marked as implemented https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | |
| 19:17:53 | dansmith | i.e. migrate twice instead of boot thrice | |
| 19:18:01 | artom | It's about forcing the migration to move the server to a cell that as different CPU numbers | |
| 19:18:29 | sean-k-mooney | you are trying to force the xml to be regenerated | |
| 19:18:33 | artom | Yeah | |
| 19:18:50 | dansmith | artom: right, so two migrations will cover the bases of migrating to an empty host which could take anything and to a host where it can't (as long as it didn't choose the same by luck) | |
| 19:18:56 | artom | The idea is to have a server with 2 NUMA cells: 2 CPUs and 3 CPUs | |
| 19:19:02 | artom | And another with just a single cell: 2 CPUs | |
| 19:19:15 | sean-k-mooney | so why not use the host/hypervioer_host name parmaters on server create | |
| 19:19:30 | artom | But such that the 2-CPU cells have different physical CPUs in them | |
| 19:19:38 | sean-k-mooney | it will allow you slect the host and it will use the schdler to validate it is sutable | |
| 19:19:46 | artom | So host2: [0, 1], host3: [0, 1, 2] [3, 4] | |
| 19:20:14 | artom | "Fill" the 3-CPU cell with a first instance | |
| 19:20:20 | artom | Then boot the 2-CPU instance | |
| 19:20:29 | dansmith | artom: is this for a tempest test or a functional test? | |
| 19:20:35 | artom | I don't care where it goes, as long as I live migrate it, its CPU pins have changed | |
| 19:20:38 | artom | dansmith, functional | |
| 19:21:16 | dansmith | okay, I think we have other tests that migrate twice just to make sure we hit both hosts, regardless of the ordering | |
| 19:21:27 | dansmith | which I prefer to modifying the scheduler, if it will cover us | |
| 19:21:45 | dansmith | targeting the build works too I guess, but it's also not the more realistic case you're trying to replicate | |
| 19:21:54 | sean-k-mooney | we currently dont have any live migration test the suceed and use the fakelibvirt dirver in teh functional tests | |
| 19:22:08 | artom | Yey, I'm first :( | |
| 19:22:32 | sean-k-mooney | so ignrogn the lack of live migration tests i dont see what the big deal is | |
| 19:22:56 | sean-k-mooney | you stare each compute node with a different config with non overlaping cpus | |
| 19:23:03 | sean-k-mooney | in the dedicate_cpu_set | |
| 19:23:10 | sean-k-mooney | then spawn a singel vm | |
| 19:23:17 | sean-k-mooney | and migrated to the ohter node | |
| 19:23:23 | sean-k-mooney | and your done | |
| 19:23:39 | artom | Uhh, literally *ALL* tests are failing with "keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL" | |