| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-07 | |||
| 17:32:53 | dansmith | idk | |
| 17:32:57 | mriedem | heh same here | |
| 17:38:17 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Only attempt a rebuild claim for an evacuation to a new host https://review.openstack.org/550545 | |
| 17:41:33 | sean-k-mooney | mriedem: there are two things that mess with pci device for migration. one we dont update allcoation correctly and 2 we dont regenerate the xml for that pci passhtrough section. | |
| 17:42:14 | sean-k-mooney | mriedem: that not a new thing though livemigration with pci devices never worked | |
| 17:42:45 | mriedem | artom is going to fix it | |
| 17:42:49 | mriedem | or die trying | |
| 17:43:09 | mriedem | sean-k-mooney: what do you mean by "don't update allocation correctly"? | |
| 17:43:12 | mriedem | claims? | |
| 17:43:22 | sean-k-mooney | mriedem: yes | |
| 17:43:23 | mriedem | or something in the pci tracker? | |
| 17:43:26 | mriedem | that's all voodoo to me | |
| 17:43:34 | sean-k-mooney | well the pci device table in the db | |
| 17:43:53 | sean-k-mooney | when we migrate we dont claim the device on the new node if i remeber correctly | |
| 17:44:10 | sean-k-mooney | and i dont think we release it on the old node either | |
| 17:45:34 | sean-k-mooney | if someone does fix that then awsome but i basically equate pci passthough = no migration even though we can make it work if we really want | |
| 17:46:26 | mriedem | maybe i'm thinking about pinned cpus | |
| 17:46:28 | mriedem | L226 https://etherpad.openstack.org/p/nova-ptg-rocky-placement | |
| 17:47:05 | sean-k-mooney | pinned cpus has a similar issue in that we dont regenerate the xml to account for the available cpus on the destination | |
| 17:47:29 | mriedem | right i think that's what artom is going to spec up and fix | |
| 17:47:44 | sean-k-mooney | mriedem: artom or stephenfin ? | |
| 17:47:47 | mriedem | artom | |
| 17:48:25 | mriedem | i guess live migrating instances with pci devices will wait for nested RPs (or something) | |
| 17:48:37 | sean-k-mooney | that would be nice to fix too. i think cold migrate works today but just not live migrate | |
| 17:49:18 | sean-k-mooney | mriedem: ya or after the multiple port binding work is done perhaps. | |
| 17:49:49 | sean-k-mooney | i think cold migrate might work with pci passhtouhg though not sure. | |
| 17:49:51 | mriedem | the cpu pinning spec is going to follow the port binding template | |
| 17:50:32 | sean-k-mooney | mriedem: for the fallback and version check in the conductor to ensure both nodes support it | |
| 17:50:45 | mriedem | yes something like that | |
| 17:50:55 | mriedem | or that the requested cpus to pin are available on the dest | |
| 17:50:58 | mriedem | and/or | |
| 17:51:05 | mriedem | if not, fail that host and try another one | |
| 17:51:51 | sean-k-mooney | ya that makes sense however with the other placement discusstion about modeling dedicated cpus as a new inventory placement can gurantee that for us | |
| 17:52:15 | mriedem | is there anything placement *can't* do? | |
| 17:52:29 | mriedem | i wish placement would have my babies | |
| 17:52:30 | sean-k-mooney | well we will still need the numa toplology filter i guest but we should be able to confirm the cpus are free before calling the compute node | |
| 17:53:39 | openstackgerrit | Merged openstack/nova master: Handle not found error on taking snapshot https://review.openstack.org/511074 | |
| 18:06:10 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Snapshot https://review.openstack.org/543023 | |
| 18:12:38 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Only attempt a rebuild claim for an evacuation to a new host https://review.openstack.org/550555 | |
| 18:22:52 | purplerbot | <mriedem> i wish placement would have my babies [2018-03-07 17:52:29.605606] [n 1nc3] | |
| 18:22:52 | cdent | I'm going to remember this one for later [t 1nc3] | |
| 18:25:30 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Fix version cap when no nova-compute started https://review.openstack.org/550557 | |
| 18:29:42 | dansmith | http://logs.openstack.org/94/550194/4/check/nova-next/67f6ca9/job-output.txt.gz#_2018-03-07_18_22_35_715619 | |
| 18:29:42 | dansmith | mriedem: do you know what this means? | |
| 18:30:33 | mriedem | bash something or other? | |
| 18:30:35 | mriedem | no | |
| 18:31:04 | mriedem | my guess would be related to something with source ${BASE}/new/devstack/lib/nova | |
| 18:31:43 | mriedem | that nova script will source the common function stuff and since we don't have stackrc sourced, GITDIR isn't defined? | |
| 18:32:05 | dansmith | yeah I bet I need to source that before nova, | |
| 18:32:13 | dansmith | else it uses gitdir as not associative | |
| 18:32:18 | dansmith | or something | |
| 18:32:24 | dansmith | however, it's doing the thing we want now | |
| 18:32:56 | mriedem | it doesn't make the script fail does it? | |
| 18:32:56 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Only attempt a rebuild claim for an evacuation to a new host https://review.openstack.org/550560 | |
| 18:33:26 | dansmith | mriedem: well, it's failing so I dunno what else caused it | |
| 18:33:38 | mriedem | i mean, if it's just noise, meh? we could set it to GITDIR=$BASE/nova if we cared | |
| 18:33:56 | dansmith | "*** FAILED with status: 1" | |
| 18:34:00 | dansmith | job failed | |
| 18:34:22 | mriedem | yeah http://logs.openstack.org/94/550194/4/check/nova-next/67f6ca9/logs/devstack-gate-post_test_hook.txt.gz#_2018-03-07_18_22_35_714 | |
| 18:35:15 | mriedem | this is nice v | |
| 18:35:16 | mriedem | http://logs.openstack.org/94/550194/4/check/nova-next/67f6ca9/logs/devstack-gate-post_test_hook.txt.gz#_2018-03-07_18_22_31_408 | |
| 18:35:42 | openstackgerrit | Dan Smith proposed openstack/nova master: Add simple db purge command https://review.openstack.org/550171 | |
| 18:35:42 | openstackgerrit | Dan Smith proposed openstack/nova master: Run post-test archive against cell1 https://review.openstack.org/550194 | |
| 18:35:43 | openstackgerrit | Dan Smith proposed openstack/nova master: Make nova-manage db purge take --all-cells https://review.openstack.org/550502 | |
| 18:35:43 | openstackgerrit | Dan Smith proposed openstack/nova master: Add --purge helper flag to archive_deleted_rows https://review.openstack.org/550182 | |
| 18:36:07 | dansmith | um | |
| 18:36:25 | dansmith | we don't have shadow on those, so I dunno what that means | |
| 18:36:39 | mriedem | that's the thing tssurya added, | |
| 18:36:48 | dansmith | to just delete them? | |
| 18:36:49 | mriedem | where once we archive instances, we delete the related mappings and reqspecs from the api db | |
| 18:37:08 | dansmith | I guess I didn't realize we were calling those "archived" in the output, because they can't be recovered | |
| 18:38:28 | mriedem | it's a bit misleading | |
| 18:39:00 | tssurya | I shouldn't have them displayed in the output ? | |
| 18:39:26 | mriedem | it's not a huge issue for me | |
| 18:39:52 | dansmith | yeah I don't care that much, it just triggered the wrong thing in my brain, so it could for others | |
| 18:40:33 | mriedem | if we care, we could put some words in the CLI docs https://docs.openstack.org/nova/latest/cli/nova-manage.html#nova-database | |
| 18:44:35 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: DiskAdapter parent class https://review.openstack.org/549053 | |
| 18:57:27 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/550569 | |
| 19:09:03 | mriedem | jaypipes: what's a VF in placement world? a resource class? | |
| 19:09:05 | artom | mriedem, wait, we're doing live migration with PCI devices now? I was only told about CPU pinning | |
| 19:09:18 | mriedem | artom: you're right, i was conflating the two earlier | |
| 19:09:50 | artom | mriedem, also, I'm pretty sure we need loads of underlying stuff for PCI devices to be live-migratable | |
| 19:10:04 | mriedem | jaypipes: or do we have CN RP (root) -> PF RP (child) -> VF RP (grandchild)? | |
| 19:10:08 | artom | We had an internal call this morning and someone brought up live migration with SRIOV | |
| 19:10:37 | artom | And while I still need to do research and ask around, the conclusion was "we need low level stuff, possible even at the kernel level" | |
| 19:11:10 | mnaser | can someone give me some pointers about where to RTFM regarding what happens between the time a new server request comes in and the vm gets a cell assigned? i'm seeing behaviour where VMs will intermittantly (under load) go in BUILD status, with no task | |
| 19:11:12 | dansmith | mriedem: VF is a resource not a provider | |
| 19:11:17 | dansmith | mriedem: a PF is a provider (of VFs) | |
| 19:11:33 | dansmith | PF is also a provider of inventory of 1 PF, in case you want to pass a PF into the guest | |
| 19:11:35 | mnaser | usually i would see BUILD status + SCHEDULING when i see any scheduler issues, but this seems before a cell gets assigned (i think?) | |
| 19:11:49 | mnaser | not looking for an answer, but maybe a pointer to where should i be reading | |
| 19:12:12 | mriedem | dansmith: ok, trying to load https://review.openstack.org/#/c/485522/ into my brain | |
| 19:13:05 | mriedem | mnaser: the build request, which represents the instance until it's created in a cell, is created in the compute API._provision_instances method | |
| 19:13:15 | mriedem | the API casts to conductor's schedule_and_build_instances method, | |
| 19:13:38 | mriedem | which calls the scheduler for hosts and then based on the host selected, finds the cell the host is in and creates the instance in that cell | |
| 19:13:41 | mriedem | then deletes the build request | |
| 19:13:53 | mnaser | mriedem: ok i see, i'll trace from there, i guess something is happening so the instance doesn't even go into scheduling state.. so i'll see what happens across there | |
| 19:14:09 | mnaser | it's strange because it just happens from time to time (but seems to happen more under stress) | |
| 19:14:47 | mriedem | dansmith: was wondering if in this trusted VF thing, with placement, if there would be a way to say a VF is trusted, but if it's a resource and not a provider, then i don't think there is | |
| 19:15:10 | mriedem | it sounds like the thing would be saying the PF has trusted VFs | |