| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 20:10:14 | openstackgerrit | Merged openstack/nova master: Suppress policy deprecated warnings in tests https://review.opendev.org/676670 | |
| 20:10:30 | artom | Except... that doens't work, because the conductor then casts to the computes to do the work | |
| 20:10:44 | artom | So if something fails in prep_resize or resize_instance, we'll never know | |
| 20:11:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 20:11:33 | efried | mriedem: ^ | |
| 20:11:36 | efried | gibi: ^ | |
| 20:15:16 | mriedem | efried: hammered you | |
| 20:16:48 | mriedem | https://youtu.be/otCpCn0l4Wo?t=110 | |
| 20:16:50 | mriedem | break it down! | |
| 20:17:19 | dansmith | artom: right, that's why I said if it happens too late | |
| 20:19:27 | openstackgerrit | Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 20:19:33 | artom | dansmith, right, just caught up with you | |
| 20:20:19 | artom | Mind you | |
| 20:22:20 | mriedem | doing a thing in the api, failing in compute, and being out of whack isn't a new problem | |
| 20:22:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418 | |
| 20:22:31 | openstackgerrit | Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991 | |
| 20:22:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946 | |
| 20:22:32 | mriedem | s/api/controller/ | |
| 20:22:32 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 20:22:48 | artom | mriedem, so do we have a "best practice" kind of thing to handle it? | |
| 20:22:56 | artom | Or it's all case by case? | |
| 20:23:18 | mriedem | you're worried that we change the request spec to use the new flavor in the api/conductor, cast to compute to do the resize, it fails and the instance is using the old flavor and the request spec is using the new flavor, right? | |
| 20:23:31 | mriedem | and then when cold migrating that server, the request spec is incorrectly using the new flavor that the instance isn't actually using | |
| 20:24:24 | artom | mriedem, yes to both | |
| 20:24:47 | artom | And really it's gibi that's worried - he brought it up on the review | |
| 20:25:16 | mriedem | there isn't really a best practice for that. the compute can't/shouldn't get an update the request spec. there is no periodic in the controller services that is healing the request spec for failed resizes. | |
| 20:25:45 | mriedem | note that in revert_resize in the API, we also update the request spec to match the source host and then cast off to do the things in compute | |
| 20:25:51 | mriedem | if those things fail, the request spec could be out of sync again | |
| 20:26:13 | mriedem | as time has gone on we've persisted less and less of the original crap we used to store in the request spec | |
| 20:26:22 | mriedem | so it was just a blob to pass things from the api to the scheduler | |
| 20:26:39 | mriedem | the flavor is tricky though since it is useful for things like down-cell api responses where we don't have the instance record | |
| 20:27:17 | mriedem | tl;dr is there a best practice for avoiding split brain? heal things periodically? use etcd? :) | |
| 20:27:21 | artom | (so what's in the api_db for instances? just a cell mapping?) | |
| 20:27:31 | mriedem | cell/instance mappings and request spec | |
| 20:27:36 | dansmith | mriedem: earlier I said it's probably legit to refresh the reqspec from the instance's flavor before we go into a scheduling operation for an existing instance | |
| 20:27:39 | mriedem | server group members | |
| 20:28:00 | dansmith | that would solve the problem I think | |
| 20:28:15 | dansmith | leaving only cell down showing some outdated instance info guesses, which wouldn't be the end of the world | |
| 20:28:16 | artom | dansmith, that's kinda where I'm leaning - if we take the definition of request spec to be what mriedem said "pass scheduling-related things from API to scheduler" | |
| 20:28:28 | dansmith | artom: yup | |
| 20:28:53 | artom | Then yeah, "load it up from latest instance makes sense' | |
| 20:29:20 | dansmith | there's another much more edge case where that makes sense too, | |
| 20:29:24 | artom | Although that makes one wonder "what's the point of request_spec in the first place, if everything it in instance anyways" | |
| 20:29:38 | dansmith | which is you restore either database from a non-coordinated backup after a disaster or failed upgrade | |
| 20:29:57 | dansmith | artom: you mean reqspec.flavor? | |
| 20:30:09 | dansmith | artom: before an instance exists, that's where we hold the flavor, | |
| 20:30:11 | artom | dansmith, yeah, and request_spec.numa_topology | |
| 20:30:11 | mriedem | artom: it's not all instance | |
| 20:30:21 | mriedem | scheduler hints, forced hosts/nodes, etc | |
| 20:30:24 | dansmith | artom: and if the cell is down, then we use the stuff in reqspec as a cache | |
| 20:30:26 | mriedem | requested destinations | |
| 20:30:45 | artom | I see | |
| 20:30:46 | dansmith | mriedem: right, hence why I asked, I assume he meant the duplicative parts of reqspec, which is just flavor, AFAIK | |
| 20:30:54 | artom | dansmith, numa_topology | |
| 20:31:01 | mriedem | it's an objectified and persisted version of the old filter_properties stuff, and some of it getting persisted always has caused a lot of problems | |
| 20:31:13 | dansmith | artom: I thought the instance numa topo ended up getting fleshed out by the virt driver, | |
| 20:31:26 | dansmith | artom: where reqspec's copy was kinda the planned topo, which would still be valid to keep I tink | |
| 20:31:37 | artom | dansmith, so there's 2 things there: 1 is the instance numa topology, which can be got from flavor and image meta | |
| 20:31:38 | mriedem | there are a few things, numa topo, flavor, image, az, pci requests | |
| 20:31:49 | artom | 2 is how it fits on a host - *that* part is virt driver | |
| 20:31:58 | artom | 1 is in reqest spec | |
| 20:32:01 | dansmith | mriedem: yeah, fair | |
| 20:32:12 | artom | Unfortunately, they're muddled in the same object :( | |
| 20:32:35 | dansmith | well, regardless, | |
| 20:32:48 | dansmith | some duplication gives us a bit of a backup in the case of cell downage as I said | |
| 20:33:00 | artom | Fair enough | |
| 20:33:39 | artom | OK, gives me stuff to think about | |
| 20:33:59 | dansmith | mriedem: doesn't pci_requests in the instance get fleshed out more than in the reqspec too? | |
| 20:34:59 | artom | I'll also need to think about backportability | |
| 20:35:13 | mriedem | maybe, don't know off the top of my head | |
| 20:35:17 | artom | Because while https://bugzilla.redhat.com/show_bug.cgi?id=1715240 is Newton is at last EOL for HR | |
| 20:35:17 | openstack | bugzilla.redhat.com bug 1715240 in openstack-nova "Resize ignores mem_page_size in new flavor" [High,On_dev] - Assigned to alifshit | |
| 20:35:17 | artom | *RH | |
| 20:35:36 | mriedem | i want to say that's Instance.pci_devices which is the allocated devices on the node | |
| 20:35:38 | artom | We'll probably want it in Queens as well | |
| 20:36:03 | artom | Need to bounce early to pick up 1/2 of my kids | |
| 20:36:06 | artom | Thank you gentlemen | |
| 20:36:12 | dansmith | mriedem: I thought we updated some physnet stuff in instance_pci_requests at least, but anyway, doesn't matter | |
| 20:36:21 | mriedem | what happened to the other half of the children? | |
| 20:36:36 | artom | mriedem, I have the other half of my marriage for that :D | |
| 20:37:05 | dansmith | I assume that when you get divorced and you have an even number of children, you just split those like the bank and other assets/liabilities right? | |
| 20:38:04 | artom | No, the woman takes everything | |
| 20:38:08 | dansmith | oh, okay | |
| 20:38:24 | artom | And you're just left with alcoholism and depresseion | |
| 20:38:42 | artom | (And those are not the names of the kids, btw) | |
| 20:38:45 | dansmith | assuming those aren't the names of half your children.. gotcha | |
| 20:38:45 | dansmith | haha | |
| 20:39:08 | artom | :D | |
| 21:01:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: ironic: report a custom trait for the node owner https://review.opendev.org/697184 | |
| 21:16:14 | eandersson | > Unable to submit allocation for instance x (409 {"errors": [{"status": 409, "request_id": "reqz", "code": "placement.undefined_code", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider y'. The requested amount would exceed the | |
| 21:16:14 | eandersson | capacity. ", "title": "Conflict"}]}) | |
| 21:16:23 | eandersson | This was the cause of the latest instance stuck in BUILD | |
| 21:17:51 | mriedem | those instances should all be buried in cell0 in conductor | |
| 21:17:56 | mriedem | scheduler should raise a NoValidHost | |
| 21:18:04 | mnaser | does setting `hw:mem_page_size` = `any` enable memory tracking on the host inside nova? reading the code leads to show that it does | |
| 21:18:28 | mriedem | eandersson: NoValidHost should be handled by conductor here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1418 | |
| 21:18:35 | mnaser | i'm running into an issue where doing numa node pinning without `hw:mem_page_size` yields in nova packing too many instances into numa node 0, which then results in `oom-killer` | |
| 21:19:06 | eandersson | I see one more log line on that req-id in the scheduler | |
| 21:19:07 | eandersson | > Computed NUMA topology CPU pinning: usable pCPUs: [[18, 38]], vCPUs mapping: [(0, 18), (1, 38)] | |
| 21:19:15 | mriedem | eandersson: and all of those build requests should result in creating instances in ERROR status in cell0 https://github.com/openstack/nova/blob/1c2b7d8f01814adfd6d28b97013a40cca51dfbdf/nova/conductor/manager.py#L1348 | |