| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 20:26:39 | mriedem | the flavor is tricky though since it is useful for things like down-cell api responses where we don't have the instance record | |
| 20:27:17 | mriedem | tl;dr is there a best practice for avoiding split brain? heal things periodically? use etcd? :) | |
| 20:27:21 | artom | (so what's in the api_db for instances? just a cell mapping?) | |
| 20:27:31 | mriedem | cell/instance mappings and request spec | |
| 20:27:36 | dansmith | mriedem: earlier I said it's probably legit to refresh the reqspec from the instance's flavor before we go into a scheduling operation for an existing instance | |
| 20:27:39 | mriedem | server group members | |
| 20:28:00 | dansmith | that would solve the problem I think | |
| 20:28:15 | dansmith | leaving only cell down showing some outdated instance info guesses, which wouldn't be the end of the world | |
| 20:28:16 | artom | dansmith, that's kinda where I'm leaning - if we take the definition of request spec to be what mriedem said "pass scheduling-related things from API to scheduler" | |
| 20:28:28 | dansmith | artom: yup | |
| 20:28:53 | artom | Then yeah, "load it up from latest instance makes sense' | |
| 20:29:20 | dansmith | there's another much more edge case where that makes sense too, | |
| 20:29:24 | artom | Although that makes one wonder "what's the point of request_spec in the first place, if everything it in instance anyways" | |
| 20:29:38 | dansmith | which is you restore either database from a non-coordinated backup after a disaster or failed upgrade | |
| 20:29:57 | dansmith | artom: you mean reqspec.flavor? | |
| 20:30:09 | dansmith | artom: before an instance exists, that's where we hold the flavor, | |
| 20:30:11 | mriedem | artom: it's not all instance | |
| 20:30:11 | artom | dansmith, yeah, and request_spec.numa_topology | |
| 20:30:21 | mriedem | scheduler hints, forced hosts/nodes, etc | |
| 20:30:24 | dansmith | artom: and if the cell is down, then we use the stuff in reqspec as a cache | |
| 20:30:26 | mriedem | requested destinations | |
| 20:30:45 | artom | I see | |
| 20:30:46 | dansmith | mriedem: right, hence why I asked, I assume he meant the duplicative parts of reqspec, which is just flavor, AFAIK | |
| 20:30:54 | artom | dansmith, numa_topology | |
| 20:31:01 | mriedem | it's an objectified and persisted version of the old filter_properties stuff, and some of it getting persisted always has caused a lot of problems | |
| 20:31:13 | dansmith | artom: I thought the instance numa topo ended up getting fleshed out by the virt driver, | |
| 20:31:26 | dansmith | artom: where reqspec's copy was kinda the planned topo, which would still be valid to keep I tink | |
| 20:31:37 | artom | dansmith, so there's 2 things there: 1 is the instance numa topology, which can be got from flavor and image meta | |
| 20:31:38 | mriedem | there are a few things, numa topo, flavor, image, az, pci requests | |
| 20:31:49 | artom | 2 is how it fits on a host - *that* part is virt driver | |
| 20:31:58 | artom | 1 is in reqest spec | |
| 20:32:01 | dansmith | mriedem: yeah, fair | |
| 20:32:12 | artom | Unfortunately, they're muddled in the same object :( | |
| 20:32:35 | dansmith | well, regardless, | |
| 20:32:48 | dansmith | some duplication gives us a bit of a backup in the case of cell downage as I said | |
| 20:33:00 | artom | Fair enough | |
| 20:33:39 | artom | OK, gives me stuff to think about | |
| 20:33:59 | dansmith | mriedem: doesn't pci_requests in the instance get fleshed out more than in the reqspec too? | |
| 20:34:59 | artom | I'll also need to think about backportability | |
| 20:35:13 | mriedem | maybe, don't know off the top of my head | |
| 20:35:17 | artom | *RH | |
| 20:35:17 | openstack | bugzilla.redhat.com bug 1715240 in openstack-nova "Resize ignores mem_page_size in new flavor" [High,On_dev] - Assigned to alifshit | |
| 20:35:17 | artom | Because while https://bugzilla.redhat.com/show_bug.cgi?id=1715240 is Newton is at last EOL for HR | |
| 20:35:36 | mriedem | i want to say that's Instance.pci_devices which is the allocated devices on the node | |
| 20:35:38 | artom | We'll probably want it in Queens as well | |
| 20:36:03 | artom | Need to bounce early to pick up 1/2 of my kids | |
| 20:36:06 | artom | Thank you gentlemen | |
| 20:36:12 | dansmith | mriedem: I thought we updated some physnet stuff in instance_pci_requests at least, but anyway, doesn't matter | |
| 20:36:21 | mriedem | what happened to the other half of the children? | |
| 20:36:36 | artom | mriedem, I have the other half of my marriage for that :D | |
| 20:37:05 | dansmith | I assume that when you get divorced and you have an even number of children, you just split those like the bank and other assets/liabilities right? | |
| 20:38:04 | artom | No, the woman takes everything | |
| 20:38:08 | dansmith | oh, okay | |
| 20:38:24 | artom | And you're just left with alcoholism and depresseion | |
| 20:38:42 | artom | (And those are not the names of the kids, btw) | |
| 20:38:45 | dansmith | haha | |
| 20:38:45 | dansmith | assuming those aren't the names of half your children.. gotcha | |
| 20:39:08 | artom | :D | |
| 21:01:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: ironic: report a custom trait for the node owner https://review.opendev.org/697184 | |
| 21:16:14 | eandersson | capacity. ", "title": "Conflict"}]}) | |
| 21:16:14 | eandersson | > Unable to submit allocation for instance x (409 {"errors": [{"status": 409, "request_id": "reqz", "code": "placement.undefined_code", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider y'. The requested amount would exceed the | |
| 21:16:23 | eandersson | This was the cause of the latest instance stuck in BUILD | |
| 21:17:51 | mriedem | those instances should all be buried in cell0 in conductor | |
| 21:17:56 | mriedem | scheduler should raise a NoValidHost | |
| 21:18:04 | mnaser | does setting `hw:mem_page_size` = `any` enable memory tracking on the host inside nova? reading the code leads to show that it does | |
| 21:18:28 | mriedem | eandersson: NoValidHost should be handled by conductor here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1418 | |
| 21:18:35 | mnaser | i'm running into an issue where doing numa node pinning without `hw:mem_page_size` yields in nova packing too many instances into numa node 0, which then results in `oom-killer` | |
| 21:19:06 | eandersson | I see one more log line on that req-id in the scheduler | |
| 21:19:07 | eandersson | > Computed NUMA topology CPU pinning: usable pCPUs: [[18, 38]], vCPUs mapping: [(0, 18), (1, 38)] | |
| 21:19:15 | mriedem | eandersson: and all of those build requests should result in creating instances in ERROR status in cell0 https://github.com/openstack/nova/blob/1c2b7d8f01814adfd6d28b97013a40cca51dfbdf/nova/conductor/manager.py#L1348 | |
| 21:19:39 | mriedem | check your conductor logs | |
| 21:19:55 | eandersson | conductor logs are empty | |
| 21:19:59 | sean-k-mooney | mnaser: just got back. hw:mem_page_size=any was intened to allow the image to choose if hugepage or small pages should be used | |
| 21:20:18 | sean-k-mooney | mnaser: i belive it should enable the numa tracking but its the one policy that is least used | |
| 21:20:23 | mnaser | sean-k-mooney: but what if i dont want neither and i just want nova to be aware of memory per numa node? | |
| 21:20:39 | mnaser | cause that involves rebooting the machine and what not to enable those | |
| 21:20:51 | eandersson | and when I say empty, there are literately no log lines over the last 12 hours. | |
| 21:21:10 | mnaser | sean-k-mooney: according to virsh capabilities, the cell only has pages with size='4' | |
| 21:21:12 | sean-k-mooney | well you cant change an exitisng image to any with out a resize | |
| 21:21:24 | mnaser | right but for newly booted instances flavor change | |
| 21:21:48 | mnaser | update th flavor extra_specs and newly booted instances will do the right thing(tm) | |
| 21:21:50 | sean-k-mooney | right so hw:mem_page_size=small would use 4k pages | |
| 21:22:18 | eandersson | btw placement failed 6 times with the same erroor before it pretty much just stopped trying | |
| 21:22:19 | sean-k-mooney | any should also use 4k pages but would allow the image to request 2mb | |
| 21:22:42 | mnaser | sean-k-mooney: ok cool, i see, so 'any' should technically make things work and make nova start tracking memory | |
| 21:22:55 | eandersson | Its probably RabbitMQ issues again, but we tested all queues and binding and none are failing. | |
| 21:22:58 | eandersson | Like we have seen in the past. | |
| 21:23:15 | sean-k-mooney | i need to triple check it but if hw:mem_page_size is defiend it is ment to enable the numa aware tracking | |
| 21:23:27 | sean-k-mooney | no mater what value you set it to | |
| 21:23:31 | mnaser | sean-k-mooney: ok ill check and report back, but my notes seem to add up to yours :> | |
| 21:23:53 | eandersson | The most frustrating part is that there are no logs, and nothing in the database, so troubleshooting these usually requires looking into the database. | |
| 21:24:04 | eandersson | *nothing from the api | |
| 21:24:07 | mnaser | eandersson: did you have any rabbitmq outage/issue? | |
| 21:24:19 | sean-k-mooney | the reason im a little less confident with hw:mem_page_size=any is i know we dont have tempest testing for it | |
| 21:24:29 | sean-k-mooney | where as we did for large and small | |
| 21:24:47 | eandersson | mnaser in the past yea and we found bad bindings after that | |
| 21:24:48 | mnaser | in this case the only reason i care about mem page size is to get nova to track memory so maybe ill try small after all | |
| 21:24:59 | eandersson | we found a way to diagnose that | |
| 21:25:10 | mnaser | eandersson: in my experience anytime rabbitmq suffers .. anything .. you end up in that weird state and you have to restart the cloud | |
| 21:25:11 | eandersson | by pushing a fake message to the compute queues | |