| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-14 | |||
| 13:05:37 | luyao | dansmith: got it! See you in an hour. :) | |
| 13:06:57 | mriedem | well here i am | |
| 13:08:55 | sean-k-mooney | mriedem: ill adress your vPMU feed back later today. thanks for reviewing it before you dropped off yesterday. | |
| 13:10:34 | mriedem | yw | |
| 13:23:08 | mriedem | sean-k-mooney: more lxc failures https://logs.opendev.org/24/676024/5/experimental/nova-lxc/9c06394/controller/logs/screen-n-cpu.txt.gz#_Aug_13_23_16_20_191786 | |
| 13:26:20 | mriedem | https://github.com/lxc/lxc/issues/1057 sounds similar | |
| 13:32:26 | sean-k-mooney | fun. debian does not symlink /sbin to /usr/sbin so perhaps its in /usr/sbin/init | |
| 13:43:21 | efried | stephenfin: That's an interesting one (your reshape chicken/egg) | |
| 13:50:39 | sean-k-mooney | stephenfin: by the way regardign the reshap you code needs to be able to handel a reshap on non upgrade cases as well. | |
| 13:50:55 | sean-k-mooney | e.g. i need to handel the reshap when then new config options are defiend | |
| 13:51:33 | sean-k-mooney | so you could deploy with train and have no config options set. then set them after and restart the compute agent whic would trigger teh reshape | |
| 13:51:48 | stephenfin | sean-k-mooney: Would you actually have a reshape there? | |
| 13:52:01 | stephenfin | surely you'd just start reporting the new VCPU/PCPU resources | |
| 13:52:07 | stephenfin | no different to changing vcpu_pin_set today | |
| 13:52:19 | stephenfin | you're not changing allocations from one type to the other | |
| 13:52:30 | sean-k-mooney | well if you were previously using cpu pinning on that host | |
| 13:52:42 | efried | We've said until now that we don't want to reshape except on upgrades. (Though I've been skeptical that we would be able to stick to that.) | |
| 13:52:54 | sean-k-mooney | then defiend the cpu_dedicated_set only it would change form a vcpu allocation to pcpu allocation | |
| 13:53:21 | stephenfin | sean-k-mooney: Nope. Remember, vcpu_pin_set is used for VCPU _and_ PCPU | |
| 13:53:45 | sean-k-mooney | yes but you dont need to enable the prefilter | |
| 13:54:18 | stephenfin | Is that related? | |
| 13:54:22 | sean-k-mooney | or are we doing the convertion to resouce:pCPUs somewere else | |
| 13:54:28 | stephenfin | efried: I _think_ I'm okay, actually | |
| 13:54:33 | sean-k-mooney | in the scheduler utils maybe? | |
| 13:55:08 | stephenfin | It seems the way the reshaping is done is that we build the list of inventory that we're going to report from the virt driver, but before we do the actual update we check for old allocations | |
| 13:56:06 | sean-k-mooney | this is what might require the reshape https://review.opendev.org/#/c/671801/19/nova/conf/workarounds.py | |
| 13:56:22 | stephenfin | So I should be able to simply check "does this compute node have any PCPU resources recorded for itself and does it have any pinned instances" and if there's a mismatch then I reshape | |
| 13:56:37 | sean-k-mooney | but i guess since you would have had to install with disable_legacy_pinning_policy_translation=true | |
| 13:56:50 | sean-k-mooney | i guess its fair to say if you are disabling that you need to reshape | |
| 13:57:08 | sean-k-mooney | you are right about both inventories being reported | |
| 13:57:10 | stephenfin | sean-k-mooney: I'm kind of lost, tbh | |
| 13:57:18 | stephenfin | That will only affect the scheduler | |
| 13:57:49 | sean-k-mooney | it affect teh plamcent query | |
| 13:58:08 | stephenfin | Right, but it won't have any impact on the compute node side | |
| 13:58:35 | stephenfin | I mean, unless you're suggesting we'd want to delay reshaping if that option was configured | |
| 13:58:40 | stephenfin | But that's not the idea | |
| 13:58:52 | sean-k-mooney | with disable_legacy_pinning_policy_translation=true flavor.vcpus is translated to resouces.VCPU when hw:cpu_ploicy=dedicated | |
| 13:59:23 | stephenfin | Yeah. So if that's the case, we continue to be able to use all the pre-Train compute nodes that aren't reporting PCPUs | |
| 13:59:36 | sean-k-mooney | yep | |
| 13:59:38 | stephenfin | But we would not be able to use any of the Train nodes that *are* reporting them | |
| 13:59:49 | sean-k-mooney | when that is removed new vms woudl be translated correctly. | |
| 13:59:54 | stephenfin | Correct | |
| 14:00:00 | sean-k-mooney | i guess old vms would be fixed when they are migrated | |
| 14:00:19 | sean-k-mooney | so ya i guess its not too much of an issue | |
| 14:00:29 | stephenfin | Their allocations will also be fixed when the compute node is upgrade | |
| 14:00:34 | stephenfin | *upgraded | |
| 14:00:44 | sean-k-mooney | you could avoid the reshape by migrating off all vms on old nodes to new ones | |
| 14:01:02 | sean-k-mooney | right | |
| 14:01:15 | stephenfin | I could, but I think that shuffling on instances between hosts has been rejected | |
| 14:01:21 | stephenfin | *of instances | |
| 14:01:33 | sean-k-mooney | ok then ya ignore me | |
| 14:02:08 | sean-k-mooney | well operators can suffle the instance. nova wont do it automatically | |
| 14:02:52 | stephenfin | So my current algorithm is if the host has no PCPU inventory *and* 'ComputeNode.numa_topology.cells[*].pinned_cpus' is set (i.e. there are some pinned instances on the host), reshape | |
| 14:03:16 | stephenfin | and reshaping involves identifying the instances on the host that are pinned and migrating their VCPU allocations wholesale to PCPU allocations | |
| 14:03:32 | sean-k-mooney | ya | |
| 14:03:38 | sean-k-mooney | i think that should work correctly | |
| 14:04:01 | stephenfin | I sadly can't reshape every consumers' allocations since there's a chance, however remote, that someone hasn't listened to us and has pinned and unpinned instances on the same host | |
| 14:04:05 | sean-k-mooney | its an online data migration? | |
| 14:04:15 | stephenfin | Yeah, run on startup | |
| 14:04:29 | stephenfin | So this code will trigger once in the entire life of the node | |
| 14:04:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 14:04:38 | stephenfin | We can theoretically remove it in U | |
| 14:05:13 | sean-k-mooney | im just thinkin about the FFU implications of that which are that we must start teh agent | |
| 14:05:18 | stephenfin | I have the code rewritten to implement that and am just finishing tests off | |
| 14:05:26 | sean-k-mooney | e.g. you cant FFU form queens to U | |
| 14:05:34 | stephenfin | Do we not do that anyway? | |
| 14:05:35 | sean-k-mooney | unless you stop at train to start the agent | |
| 14:06:00 | stephenfin | If not, that might justify us keeping the reshape code around for a few cycles | |
| 14:06:21 | stephenfin | bbiab | |
| 14:06:22 | sean-k-mooney | its only the removal of the resahpe code that would break ffu i think | |
| 14:06:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 14:06:49 | sean-k-mooney | or rather require it to start the agents on an intermendiate release | |
| 14:10:12 | luyao | dansmith: Are you around now? | |
| 14:10:49 | dansmith | luyao: almost done | |
| 14:12:23 | spatel | sean-k-mooney: Do you know what is going on here? http://paste.openstack.org/show/757000/ | |
| 14:12:41 | luyao | dansmith: Great, I'll give my question first. It's about libvirt driver, it seems that it's not recommended to access DB in libvirt driver, but in my patch, I need to get flavor id from db to populate the device manager, is that acceptable? I would like you help to give some comments. | |
| 14:12:43 | spatel | It clearly related to resize issue or bug | |
| 14:13:42 | sean-k-mooney | ya i have seen that before. | |
| 14:13:54 | sean-k-mooney | are you deploying on nfs or shared stoage | |
| 14:14:04 | dansmith | luyao: no, you can't access the database directly, but the flavor is on the instance so it should already be there and have what you need | |
| 14:15:15 | sean-k-mooney | spatel: this can happen when an instance was delete in the db when the compute agent is stopped and then was archived before it was started again | |
| 14:15:50 | sean-k-mooney | e.g. after an evacutation | |
| 14:16:10 | luyao | dansmith: https://review.opendev.org/#/c/672957/5/nova/virt/libvirt/device.py@196 I need to access db to get instance | |
| 14:16:35 | sean-k-mooney | the host has a stale libvirt domain that references a disk tha that is nolonger found | |
| 14:16:39 | dansmith | luyao: you've got it right there | |
| 14:16:43 | spatel | sean-k-mooney: you are saying someone delete instance but compute agent was down and it got out of sync. | |
| 14:17:29 | dansmith | lyarwood: Instance.get_by_uuid() is okay, it accesses the database, but not directly, through conductor | |
| 14:17:34 | sean-k-mooney | yes basically this happens becasue a statle libvirt domain xml is on the host that references an image file that nolonger exists | |
| 14:17:38 | dansmith | oops | |
| 14:17:39 | spatel | I believe this is related to resize issue when you do CPU pinning which left resize stall | |
| 14:17:43 | dansmith | luyao: ^ | |
| 14:18:13 | luyao | dansmith: so it's ok to access database like this? I just thought it's not recommended to access database in driver | |
| 14:18:15 | dansmith | luyao: but you should try to avoid that if you can pull the instance there.. I'd have to look at the code to see where this is running | |
| 14:18:35 | dansmith | luyao: like I just said, you're not hitting the database directly with that call | |
| 14:19:19 | sean-k-mooney | spatel: it does look similar to https://bugs.launchpad.net/nova/+bug/1774249 | |
| 14:19:19 | openstack | Launchpad bug 1774249 in OpenStack Compute (nova) pike "update_available_resource will raise DiskNotFound after resize but before confirm" [Medium,Triaged] | |
| 14:19:22 | spatel | sean-k-mooney: do you know how do i fix this issue or clean up left over disk. | |
| 14:20:01 | dansmith | luyao: see there are now two places in that init routine that get instances, the mdev one and your pmem one | |
| 14:20:29 | dansmith | luyao: the mdev one is more efficient than what you have, but it's not okay to look up all the instances twice, so you need to refactor that code in a patch ahead of time, | |
| 14:20:31 | alex_xu | dansmith: yea, that code is running when the nova-compute start, we populate all the assigned vgpu or vpmems from the libvirt, then we need instance_uuid and flavor_id to identify each assignment | |