| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-15 | |||
| 19:35:39 | dansmith | no, we definitely don't want that | |
| 19:35:40 | mnaser | (for the future, but backport just fixing the behaviour in quota.py) | |
| 19:36:03 | mnaser | im sure there's details im missing why its not a good idea :) | |
| 19:36:07 | dansmith | if a caller is looking for a set of instance mappings, you want it to get what it asked for, not altered by some random config flag | |
| 19:37:54 | mnaser | yep makes sense | |
| 19:38:12 | mnaser | the patch to fix this seems trivial, so ill push something up | |
| 19:38:17 | dansmith | that's a very low-level database query routine.. the "should we optimize for lots of cells or not" is a very high-level decision | |
| 19:38:38 | dansmith | mnaser: cool, add me and melwitt, but the latter is the important review I thnk | |
| 19:39:50 | mnaser | dansmith: will do. i'll write up a bug right now too | |
| 19:40:02 | dansmith | thanks | |
| 19:40:06 | mnaser | dansmith: i could also workaronud this by enabling quota from placement, but yeah | |
| 19:40:08 | mnaser | let's get it fixed | |
| 19:40:22 | dansmith | mnaser: you could also archive your damned database now and then :D | |
| 19:40:32 | mnaser | dansmith: haha, always me bringing up those massive db issues :p | |
| 19:40:57 | dansmith | I guess the benefit of not archiving is giving us some data on these things, so ... your choice :D | |
| 19:53:27 | mnaser | it's friday and i did a bad job at writing this probably: https://bugs.launchpad.net/nova/+bug/1878979 | |
| 19:53:27 | openstack | Launchpad bug 1878979 in OpenStack Compute (nova) "Quota code does not respect [api]/instance_list_per_project_cells" [Undecided,New] | |
| 20:08:59 | mriedem | mnaser: related https://bugs.launchpad.net/nova/+bug/1857306 | |
| 20:08:59 | openstack | Launchpad bug 1857306 in OpenStack Compute (nova) "_bury_in_cell0 could not handle instance duplicate exception" [Low,Triaged] | |
| 20:09:10 | mriedem | wait a minute, that's not right | |
| 20:09:34 | mriedem | https://review.opendev.org/#/c/700456/ | |
| 20:12:38 | mnaser | mriedem: ehhhh, that's a lot to think about for friday :( | |
| 20:13:46 | mriedem | you're stuck inside, what else do you have to do | |
| 20:37:23 | mnaser | welp, _instances_cores_ram_count_legacy has zero tests | |
| 20:47:03 | mriedem | i think those are covered with functional tests | |
| 20:47:11 | mriedem | depending on if the placement flag is configured to use placement for quota or not | |
| 20:47:28 | mnaser | mriedem: yeah i found some unit test ones but they mostly mock that function out | |
| 20:47:45 | mriedem | yeah it's functional tests i'm pretty sure | |
| 20:47:53 | mriedem | more reliable for those kind of tests hitting the dbs | |
| 20:47:57 | mnaser | mriedem: ok thanks ill go dig there | |
| 20:48:24 | mnaser | mriedem: my test is mostly "if flag X is enabled then make sure you call with all cells otherwise make sure you retrieve list of cells | |
| 20:49:25 | openstackgerrit | Mohammed Naser proposed openstack/nova master: WIP: Make quotas respect instance_list_per_project_cells https://review.opendev.org/728575 | |
| 20:49:26 | mnaser | mriedem: ^ for context | |
| 20:49:54 | mnaser | yeah that code is broken now that i look at it again | |
| 20:50:23 | mnaser | ah no nvm, python lets vars defined in if get accessed outside | |
| 20:52:42 | openstackgerrit | Lin Yang proposed openstack/os-traits master: CPU: add traits for new AVX512 support https://review.opendev.org/728576 | |
| 21:15:29 | openstackgerrit | Lin Yang proposed openstack/os-traits master: CPU: add traits for new AVX512 support https://review.opendev.org/728576 | |
| 21:24:06 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Make quotas respect instance_list_per_project_cells https://review.opendev.org/728575 | |
| 21:24:16 | mnaser | dansmith, melwitt: ^ cooked up and tested locally :) thanks for the hints | |
| 21:28:36 | openstackgerrit | Gage Hugo proposed openstack/nova stable/stein: [DNM] - testing numa placement https://review.opendev.org/728579 | |
| 22:32:48 | melwitt | mnaser: here's the func test for cores ram count https://github.com/openstack/nova/blob/master/nova/tests/functional/db/test_quota.py#L142 | |
| 22:33:21 | melwitt | oh you found it. nevermind | |
| 22:35:52 | mnaser | melwitt: :D | |
| 22:36:25 | mnaser | I tried to make it as backportable as possible | |
| 22:37:00 | melwitt | good call. this doesn't look too bad at all, so thank you for that | |
| 22:38:09 | melwitt | mnaser: I assume you tried out this patch? what was the perf result, just curious | |
| 22:39:50 | ikla | I'm trying to get pci_passthrough to work and followed the documents. I keep seeing Pci claimed failed. Anyway to drill this down. Config looks perfect. | |
| 22:40:34 | mnaser | melwitt: I have not yet actually! It’s Friday and the load dropped so it wasn’t hitting as hard | |
| 22:41:07 | mnaser | I can try on Monday though but I imagine it’ll drop scanning instance mappings and I only have 2 cells really (0 and 1) | |
| 22:41:10 | melwitt | mnaser: I guess, this seems like the opposite of what you wrote in the bug report. it's always limiting to project_id and you're adding some that would look at all cells? | |
| 22:42:04 | mnaser | melwitt: sorry I’ve been pretty slow today. But yes before it was limited to project ID and this just doesn’t limit and checks all cells anywyas | |
| 22:42:27 | melwitt | like before your patch it's always only looking at a particular project_id and quota check is always scoped to a project, so there's no need to look at all cells | |
| 22:43:23 | melwitt | mnaser: if that's the case, that would mean there's a bug in objects.CellMappingList.get_by_project_id if it's not actually scoping down to a project | |
| 22:44:28 | mnaser | melwitt: that function does actually scope by project. The only problem is that to do that, it needs to scan all instance mappings to find all cells for that project which is where I took the performance hit | |
| 22:44:55 | mnaser | After my patch, nova checks all cells, regardless if a user has instances in it or not | |
| 22:45:00 | melwitt | and no worry, I've been slow today too. spent all day/night yesterday dealing with fallout from a nova gate breakage and trying to get it fixed. so exhausted | |
| 22:45:18 | mnaser | Sorry if my commit message or bug aren’t clear, I’m just feeling bleeeh and dazed all day lol | |
| 22:47:49 | melwitt | mnaser: sorry this just doesn't seem to make sense. before your patch, it's always limiting to one project cells. after you patch, it's capable of not limiting. I don't see how that would help with any instance mapping scanning? | |
| 22:48:48 | melwitt | oh hang on | |
| 22:49:08 | mnaser | melwitt: I think we may be talking about two scans. One for the quota calculation and one to find which cells the user has instances in | |
| 22:49:19 | melwitt | ok so you're saying when you can look at all cells, you actually go _faster_ because you don't do the instance mapping thing? wow ok | |
| 22:49:39 | melwitt | yeah ok it's a tradeoff, slowly remembering now | |
| 22:49:49 | mnaser | melwitt: https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L275-L285 | |
| 22:49:57 | melwitt | the project limiting behavior is actually worse for single cell-ers | |
| 22:50:00 | melwitt | yeah, I see now | |
| 22:50:01 | melwitt | sorry | |
| 22:50:11 | mnaser | All good! Yeah, it hurt me more in this case | |
| 22:50:24 | melwitt | yeah, I see | |
| 22:50:28 | melwitt | nice catch | |
| 22:50:47 | mnaser | not the catch I wanted on a Friday but it is what it is :P | |
| 22:50:54 | melwitt | haha right | |
| 22:51:13 | melwitt | ok, I'll review this properly a bit later. thanks for opening the bug and doing the patch | |
| 22:55:27 | mnaser | No worries | |
| 23:05:21 | ikla | Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? | |
| 23:05:29 | ikla | any ideas? | |
| 23:06:57 | ikla | or normal | |
| 23:07:33 | openstackgerrit | Merged openstack/nova master: docs: Add evacuation pre-conditions around the src host https://review.opendev.org/723853 | |
| 23:10:00 | melwitt | ikla: nova-conductor is a required piece of nova, it needs to be running so if it isn't, that would be a problem | |
| 23:31:59 | openstackgerrit | Merged openstack/nova master: Reproduce bug 1869050 https://review.opendev.org/714997 | |
| 23:31:59 | openstack | bug 1869050 in OpenStack Compute (nova) "migration of anti-affinity server fails due to stale scheduler instance info" [Low,In progress] https://launchpad.net/bugs/1869050 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 23:32:20 | ikla | Insufficient compute resources: Claim pci failed. - is there anyway to see why it failed? | |
| #openstack-nova - 2020-05-16 | |||
| 00:19:47 | openstackgerrit | Merged openstack/nova master: Update scheduler instance info at confirm resize https://review.opendev.org/714998 | |
| 01:36:46 | openstackgerrit | Brin Zhang proposed openstack/nova master: Bump hacking min version to 3.1.0 https://review.opendev.org/727589 | |
| 01:58:46 | openstackgerrit | Merged openstack/python-novaclient master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728270 | |
| 02:01:13 | ikla | any idea why instances would get stuck in deleting and building state? anything I can do? | |
| 02:31:07 | openstackgerrit | Merged openstack/nova master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728263 | |
| 04:25:31 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512 | |
| 04:25:51 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.add_metaclass https://review.opendev.org/726565 | |
| 04:25:59 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898 | |
| 04:26:12 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | |
| 04:26:22 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | |
| 04:26:33 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | |
| 04:26:43 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | |
| 04:26:53 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | |
| 04:27:20 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 07:30:56 | openstackgerrit | Andreas Jaeger proposed openstack/nova-specs master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728611 | |
| 07:33:44 | openstackgerrit | Andreas Jaeger proposed openstack/nova-specs master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728611 | |
| 07:34:03 | openstackgerrit | Andreas Jaeger proposed openstack/os-vif master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728612 | |
| 22:40:29 | openstackgerrit | Merged openstack/nova stable/queens: Add config option for neutron client retries https://review.opendev.org/722822 | |
| #openstack-nova - 2020-05-17 | |||
| 05:39:47 | openstackgerrit | Nalini Varshney proposed openstack/nova master: Add migration to make key field type VARBINARY in aggregate_metadata table, https://review.opendev.org/725522 | |