| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-24 | |||
| 16:25:21 | sean-k-mooney | cdent: i was in meetings so and some other stff so ill have the simple booling flag version up soon | |
| 16:26:33 | openstackgerrit | Merged openstack/nova-specs master: Re-proposes multiple vGPU types in libvirt https://review.openstack.org/602474 | |
| 16:26:34 | mdbooth | cdent: Yeah, having thought about that again, I think it would need a different oslo.db api to do that. The decorator is returned by the object we want to replace, so there's no getting round that. | |
| 16:35:42 | melwitt | bauzas: just noticed another thing for the vgpu spec follow up https://review.openstack.org/#/c/602474/2/specs/stein/approved/vgpu-stein.rst@11 | |
| 16:35:47 | melwitt | bp name | |
| 17:09:52 | openstack | bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [Undecided,Confirmed] https://launchpad.net/bugs/1799727 | |
| 17:09:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115 | |
| 17:13:29 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Update port profile unit tests in host_info https://review.openstack.org/610636 | |
| 17:14:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115 | |
| 17:14:18 | openstack | bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,Confirmed] https://launchpad.net/bugs/1799727 | |
| 17:25:36 | cfriesen | jaypipes: stephenfin: regarding the "show server numa topology" spec, are you okay with showing the *guest* topology for regular users if we clean up the various issues you raised in the spec? | |
| 17:27:00 | sean-k-mooney | cfriesen: provdied the show numa toplogy spec does not show and host topology info they i think its fine | |
| 17:27:57 | sean-k-mooney | cfriesen: if you want it to show how the virtual topology is map to a hosts phyisical toplogy then that would be admin only | |
| 17:31:28 | cfriesen | sean-k-mooney: agreed. I think showing the "expected" host details to the admin would be useful, since we've run into cases where expected didn't match actual. :) | |
| 17:33:41 | sean-k-mooney | cfriesen: it should not upstream. the intel nfv ci actully ssh's into the host that the vm is running on and validates its pinned correctly | |
| 17:34:44 | cfriesen | sean-k-mooney: live migration | |
| 17:34:47 | sean-k-mooney | cfriesen: but for an admin yes it could be useful when debuging | |
| 17:35:15 | cfriesen | (at least until the patch goes in to fail the live migration if there's a numa topology) | |
| 17:35:15 | sean-k-mooney | cfriesen: what about it i said we validated it was pinned as nova told it too | |
| 17:35:27 | sean-k-mooney | i did not say nova pinned it correctly | |
| 17:36:10 | sean-k-mooney | cfriesen: ya on that i have asked stephen to make that condional and off by default | |
| 17:36:12 | cfriesen | sean-k-mooney: we ran into some bugs during aborted/failed operations | |
| 17:37:01 | openstackgerrit | Merged openstack/nova stable/rocky: Fix up compute rpcapi version for pike release https://review.openstack.org/612561 | |
| 17:37:26 | sean-k-mooney | cfriesen: i know of at least on production largscale deployment that uses ovs-dpdk which means the guest have hugepages and numa toplogy that uses livemigration | |
| 17:38:22 | sean-k-mooney | it is still true that live migration can fail because there are not enough free hugepges on the numa node but the failure rate was low enouh that they were happy to contiue to use it | |
| 17:39:38 | sean-k-mooney | mainly since they could just specify a host that they knew would fit the instance | |
| 17:40:52 | jaypipes | cfriesen: I'm not thrilled about it, no.. | |
| 17:41:22 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 17:41:40 | sean-k-mooney | jaypipes: even if its just the computed toplogy form the flavor+image with no host info? | |
| 17:42:01 | cfriesen | jaypipes: so currently an end-user can't tell their topology without logging into the guest and checking it. if they're using a per-user keypair, other users in the same tenant can't see what the topology is. | |
| 17:42:16 | dansmith | cfriesen: what's the use case for that though? | |
| 17:42:22 | sean-k-mooney | cfriesen: well they can they can look at the flavor and image metadata | |
| 17:42:38 | cfriesen | sean-k-mooney: not all clouds allow end users to see flavor extra specs | |
| 17:43:04 | sean-k-mooney | cfriesen: wait they dont? how do you know what the falvor does without that info | |
| 17:43:48 | dansmith | sean-k-mooney: historically those were admin only | |
| 17:43:56 | cfriesen | dansmith: for an admin it's useful for showing what nova expects the virt/phys mapping to be, which can then be checked against the actual mapping on the hypervisor. | |
| 17:44:20 | sean-k-mooney | dansmith: huh ok i guess i just always am an admin so never noticed | |
| 17:44:45 | cfriesen | for a normal user, it's useful in the same way knowing how many cpus or how much ram your instance has is useful | |
| 17:44:58 | dansmith | cfriesen: you mean it's useful for an admin to make sure nova is doing the thing it expects? that seems like a weak case to me.. | |
| 17:45:37 | dansmith | cfriesen: but .. the user can get that from the guest if they're the user. I just have a hard time understanding what they can do with that info from just the API, other than maybe complain, or notice that it changed when their admin migrates them | |
| 17:46:10 | sean-k-mooney | cfriesen: is any of this stuff already in the metadata api? | |
| 17:46:26 | cfriesen | dansmith: only a user with a suitable keypair can login to the guest. other users in the same tenant can't. | |
| 17:46:36 | cfriesen | or at least might not be able to | |
| 17:46:40 | dansmith | cfriesen: right, it's that case I don't get | |
| 17:47:07 | dansmith | cfriesen: like, if I can log into the guest, what does it matter what the topology is? | |
| 17:47:39 | dansmith | I mean I can come up with completely synthetic reasons, but they're exceedingly weak, which is what I said above | |
| 17:49:19 | dansmith | sorry.. "can't log into the guest" | |
| 17:54:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Update allocation_ratios in placement inventory if config changes https://review.openstack.org/613126 | |
| 17:55:33 | cfriesen | most of the uses that we added it for were admin-level, admittedly. Like figuring out why something is having a hard time scheduling on a migration or showing the expected virt/phys mapping. For an end user it's really just about showing all the available information about the instance and making it so they don't have to jump through hoops to get it. | |
| 17:56:30 | cdent | "making it so they don't have to jump through hoops to get it" <- that ought to be compelling enough? | |
| 18:00:27 | openstackgerrit | Gaudenz Steinlin proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | |
| 18:01:34 | sean-k-mooney | cdent: its 3 command to 1 but if some clouds hide flavor extra spec then i guess maybe they cant run the 3 commands | |
| 18:01:55 | sean-k-mooney | cfriesen: had you also planned to show cpu topplogy or just numa topology | |
| 18:02:51 | sean-k-mooney | cfriesen: as we both know they can and usually are very different by defualt on openstack | |
| 18:03:16 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove unused code https://review.openstack.org/613127 | |
| 18:03:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove QuotaDriver.destroy_all_by_project() https://review.openstack.org/613129 | |
| 18:03:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove unused Quota driver methods https://review.openstack.org/613128 | |
| 18:03:18 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove default kwarg on get_class_quotas() https://review.openstack.org/613130 | |
| 18:04:29 | jaypipes | melwitt, dansmith: some cleanups of the quota system ^^ | |
| 18:05:28 | cfriesen | sean-k-mooney: we currently show memory size, page size, and which guest CPUs are associated with each guest numa node. showing virtual CPU topology (sockets/cores/threads) would be a lot trickier since that's not in the nova DB. | |
| 18:06:12 | sean-k-mooney | well it is if set in the flavor or image else it up to the virt driver | |
| 18:06:15 | jaypipes | melwitt, dansmith: more patches on the way but those are a good first chunk of removing cruft | |
| 18:06:50 | sean-k-mooney | cfriesen: so if we were going this route it would be nice to include it if set in image or flavor | |
| 18:07:36 | cfriesen | sean-k-mooney: don't we have scenarios where we say "max cpus per socket"? in that case only the virt driver knows the actual number | |
| 18:13:42 | sean-k-mooney | cfriesen: yes but we can also say 2 cpus per socket instead of max | |
| 18:13:50 | sean-k-mooney | anyway that is just a taught | |
| 18:14:37 | sean-k-mooney | it sound like dansmith and jaypipes would prefer this not to be in the api anyway so maybe you could do it as an osc or nova client feature | |
| 18:21:41 | cfriesen | dansmith: jaypipes: currently there's no way for an admin to look at the expected virt/phys mapping without going into the database. do we expect that nova admins will always have raw DB access? | |
| 18:22:11 | sean-k-mooney | they dont need db acess | |
| 18:22:29 | sean-k-mooney | they just need to be able to do a flaovr show and image show + look at teh libvirt xml | |
| 18:22:47 | cfriesen | sean-k-mooney: no, I'm talking about which specific guest vcpu maps to which specific host CPU | |
| 18:22:59 | sean-k-mooney | that in the libvirt xml | |
| 18:23:20 | cfriesen | sean-k-mooney: that's the hypervisor view, not nova's view (which in buggy cases can be different) | |
| 18:23:48 | sean-k-mooney | the database is not going to help you in does cases to debug what went wrong | |
| 18:24:13 | sean-k-mooney | we have no way to get the numa_toplogy blob from the moment when nova was caluating the pinning | |
| 18:24:22 | cfriesen | sure it can...if I can see that the entry in the database matched the previous mappings from before I did a migration... | |
| 18:25:25 | sean-k-mooney | wait your talking about migrtiaon with cpu pinning wich today is not supported | |
| 18:25:31 | cfriesen | cold migration is | |
| 18:25:48 | sean-k-mooney | cold migration yes but that wont break in this case | |
| 18:26:07 | sean-k-mooney | the xml will be regenerated on the new host | |
| 18:26:38 | cfriesen | throw in power outages and downed compute nodes and lost messages and migration reverts | |
| 18:27:31 | sean-k-mooney | so when the db is in an undefiend state it may not agree with the hypervior | |
| 18:27:50 | sean-k-mooney | yes that is true. not sure this will help with htat | |
| 18:28:13 | cfriesen | sean-k-mooney: it'll at least tell us what the problem is | |
| 18:28:37 | sean-k-mooney | the problem being the db is borked | |
| 18:29:22 | sean-k-mooney | if the vm is running it means the hypervior pinned it correctly based on the info it had at the time. | |
| 18:30:02 | sean-k-mooney | it should never be the case that the db is correct and vm is wrong in the cold migrate case | |
| 18:30:15 | sean-k-mooney | live migrate this can invert | |
| 18:31:09 | sean-k-mooney | cfriesen: would a error log message generate by one of the periodic task on the compute agent not be more useful? | |
| 18:31:40 | sean-k-mooney | e.g. dicoverd instance x with pinning y expect z | |
| 18:32:57 | openstackgerrit | sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034 | |
| 18:34:24 | sean-k-mooney | melwitt: i updated the placement wsgi patch again based on more talks with mdbooth and cdent earilier | |
| 18:34:41 | sean-k-mooney | melwitt: do you still want me to drop the second unit test https://review.openstack.org/#/c/610034/5/nova/tests/unit/api/openstack/placement/test_db_api.py | |
| 18:34:52 | sean-k-mooney | if so i can respin it quickly | |
| 18:37:07 | cfriesen | sean-k-mooney: an error log like that is not a bad idea, actually. | |
| 18:38:05 | sean-k-mooney | im kindof assuming any resonable size cloud that is going to have this problem is liekly exporting there logs to elastic serach and or similar and could set up an alert for it | |
| 18:39:32 | melwitt | sean-k-mooney: commented | |
| 18:41:10 | sean-k-mooney | melwitt: thanks | |
| 18:46:33 | cfriesen | sean-k-mooney: actually, I was wrong. we do have the actual guest CPU topology in the InstanceNUMACell, so we could display it too. | |
| 18:47:02 | sean-k-mooney | in the instance request spec im assuming | |