| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-14 | |||
| 12:42:40 | alemgeta | hello please i'm doing my msc thesis on openstack nova, what exactly failure detection algorithm used in openstack ,and place of the code, its my appreciation | |
| 12:43:49 | alemgeta | hello please i'm doing my msc thesis on openstack nova, what exactly failure detection algorithm used in openstack ,and place of the code, its my appreciation | |
| 12:55:39 | tssurya | mriedem, dansmith: are you both around ? | |
| 12:55:55 | dansmith | I am but I'm about to jump on a call in 4 minutes | |
| 12:56:20 | tssurya | what did we finally decide on ? tweaking the reno and adding the fact that config on ironic side should be diasbled ? or pushing the task state update to driver level | |
| 12:57:03 | dansmith | not sure we decided anything specific, but I'd prefer to move the task state setting out of the api | |
| 12:57:23 | tssurya | ok let me try to do that then | |
| 12:57:36 | dansmith | mriedem isn't around right now | |
| 12:58:09 | tssurya | ok I'll push the rest of his suggestions and we can come back to that when he comes online | |
| 12:58:45 | alemgeta | hello someone i have little question about openstack nova | |
| 12:59:09 | alemgeta | please someone | |
| 12:59:22 | alemgeta | about failer detection | |
| 12:59:40 | alemgeta | doing msc thesis | |
| 13:02:07 | luyao | dansmith: when will you back?😀 | |
| 13:02:11 | sean-k-mooney | nova does not perfrom failure detection of any of the workloads that are deploy with it if that is what you were wondering. it obviously validates that the action you asked it to perfom succeeded but it does not monitor the apllication lifetime | |
| 13:03:42 | sean-k-mooney | dansmith: speak his name an he shall appear | |
| 13:04:33 | sean-k-mooney | mriedem: tssurya wanted to know if you were around ~10 minuts ago | |
| 13:04:37 | dansmith | in an hour | |
| 13:05:37 | luyao | dansmith: got it! See you in an hour. :) | |
| 13:06:57 | mriedem | well here i am | |
| 13:08:55 | sean-k-mooney | mriedem: ill adress your vPMU feed back later today. thanks for reviewing it before you dropped off yesterday. | |
| 13:10:34 | mriedem | yw | |
| 13:23:08 | mriedem | sean-k-mooney: more lxc failures https://logs.opendev.org/24/676024/5/experimental/nova-lxc/9c06394/controller/logs/screen-n-cpu.txt.gz#_Aug_13_23_16_20_191786 | |
| 13:26:20 | mriedem | https://github.com/lxc/lxc/issues/1057 sounds similar | |
| 13:32:26 | sean-k-mooney | fun. debian does not symlink /sbin to /usr/sbin so perhaps its in /usr/sbin/init | |
| 13:43:21 | efried | stephenfin: That's an interesting one (your reshape chicken/egg) | |
| 13:50:39 | sean-k-mooney | stephenfin: by the way regardign the reshap you code needs to be able to handel a reshap on non upgrade cases as well. | |
| 13:50:55 | sean-k-mooney | e.g. i need to handel the reshap when then new config options are defiend | |
| 13:51:33 | sean-k-mooney | so you could deploy with train and have no config options set. then set them after and restart the compute agent whic would trigger teh reshape | |
| 13:51:48 | stephenfin | sean-k-mooney: Would you actually have a reshape there? | |
| 13:52:01 | stephenfin | surely you'd just start reporting the new VCPU/PCPU resources | |
| 13:52:07 | stephenfin | no different to changing vcpu_pin_set today | |
| 13:52:19 | stephenfin | you're not changing allocations from one type to the other | |
| 13:52:30 | sean-k-mooney | well if you were previously using cpu pinning on that host | |
| 13:52:42 | efried | We've said until now that we don't want to reshape except on upgrades. (Though I've been skeptical that we would be able to stick to that.) | |
| 13:52:54 | sean-k-mooney | then defiend the cpu_dedicated_set only it would change form a vcpu allocation to pcpu allocation | |
| 13:53:21 | stephenfin | sean-k-mooney: Nope. Remember, vcpu_pin_set is used for VCPU _and_ PCPU | |
| 13:53:45 | sean-k-mooney | yes but you dont need to enable the prefilter | |
| 13:54:18 | stephenfin | Is that related? | |
| 13:54:22 | sean-k-mooney | or are we doing the convertion to resouce:pCPUs somewere else | |
| 13:54:28 | stephenfin | efried: I _think_ I'm okay, actually | |
| 13:54:33 | sean-k-mooney | in the scheduler utils maybe? | |
| 13:55:08 | stephenfin | It seems the way the reshaping is done is that we build the list of inventory that we're going to report from the virt driver, but before we do the actual update we check for old allocations | |
| 13:56:06 | sean-k-mooney | this is what might require the reshape https://review.opendev.org/#/c/671801/19/nova/conf/workarounds.py | |
| 13:56:22 | stephenfin | So I should be able to simply check "does this compute node have any PCPU resources recorded for itself and does it have any pinned instances" and if there's a mismatch then I reshape | |
| 13:56:37 | sean-k-mooney | but i guess since you would have had to install with disable_legacy_pinning_policy_translation=true | |
| 13:56:50 | sean-k-mooney | i guess its fair to say if you are disabling that you need to reshape | |
| 13:57:08 | sean-k-mooney | you are right about both inventories being reported | |
| 13:57:10 | stephenfin | sean-k-mooney: I'm kind of lost, tbh | |
| 13:57:18 | stephenfin | That will only affect the scheduler | |
| 13:57:49 | sean-k-mooney | it affect teh plamcent query | |
| 13:58:08 | stephenfin | Right, but it won't have any impact on the compute node side | |
| 13:58:35 | stephenfin | I mean, unless you're suggesting we'd want to delay reshaping if that option was configured | |
| 13:58:40 | stephenfin | But that's not the idea | |
| 13:58:52 | sean-k-mooney | with disable_legacy_pinning_policy_translation=true flavor.vcpus is translated to resouces.VCPU when hw:cpu_ploicy=dedicated | |
| 13:59:23 | stephenfin | Yeah. So if that's the case, we continue to be able to use all the pre-Train compute nodes that aren't reporting PCPUs | |
| 13:59:36 | sean-k-mooney | yep | |
| 13:59:38 | stephenfin | But we would not be able to use any of the Train nodes that *are* reporting them | |
| 13:59:49 | sean-k-mooney | when that is removed new vms woudl be translated correctly. | |
| 13:59:54 | stephenfin | Correct | |
| 14:00:00 | sean-k-mooney | i guess old vms would be fixed when they are migrated | |
| 14:00:19 | sean-k-mooney | so ya i guess its not too much of an issue | |
| 14:00:29 | stephenfin | Their allocations will also be fixed when the compute node is upgrade | |
| 14:00:34 | stephenfin | *upgraded | |
| 14:00:44 | sean-k-mooney | you could avoid the reshape by migrating off all vms on old nodes to new ones | |
| 14:01:02 | sean-k-mooney | right | |
| 14:01:15 | stephenfin | I could, but I think that shuffling on instances between hosts has been rejected | |
| 14:01:21 | stephenfin | *of instances | |
| 14:01:33 | sean-k-mooney | ok then ya ignore me | |
| 14:02:08 | sean-k-mooney | well operators can suffle the instance. nova wont do it automatically | |
| 14:02:52 | stephenfin | So my current algorithm is if the host has no PCPU inventory *and* 'ComputeNode.numa_topology.cells[*].pinned_cpus' is set (i.e. there are some pinned instances on the host), reshape | |
| 14:03:16 | stephenfin | and reshaping involves identifying the instances on the host that are pinned and migrating their VCPU allocations wholesale to PCPU allocations | |
| 14:03:32 | sean-k-mooney | ya | |
| 14:03:38 | sean-k-mooney | i think that should work correctly | |
| 14:04:01 | stephenfin | I sadly can't reshape every consumers' allocations since there's a chance, however remote, that someone hasn't listened to us and has pinned and unpinned instances on the same host | |
| 14:04:05 | sean-k-mooney | its an online data migration? | |
| 14:04:15 | stephenfin | Yeah, run on startup | |
| 14:04:29 | stephenfin | So this code will trigger once in the entire life of the node | |
| 14:04:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | |
| 14:04:38 | stephenfin | We can theoretically remove it in U | |
| 14:05:13 | sean-k-mooney | im just thinkin about the FFU implications of that which are that we must start teh agent | |
| 14:05:18 | stephenfin | I have the code rewritten to implement that and am just finishing tests off | |
| 14:05:26 | sean-k-mooney | e.g. you cant FFU form queens to U | |
| 14:05:34 | stephenfin | Do we not do that anyway? | |
| 14:05:35 | sean-k-mooney | unless you stop at train to start the agent | |
| 14:06:00 | stephenfin | If not, that might justify us keeping the reshape code around for a few cycles | |
| 14:06:21 | stephenfin | bbiab | |
| 14:06:22 | sean-k-mooney | its only the removal of the resahpe code that would break ffu i think | |
| 14:06:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 14:06:49 | sean-k-mooney | or rather require it to start the agents on an intermendiate release | |
| 14:10:12 | luyao | dansmith: Are you around now? | |
| 14:10:49 | dansmith | luyao: almost done | |
| 14:12:23 | spatel | sean-k-mooney: Do you know what is going on here? http://paste.openstack.org/show/757000/ | |
| 14:12:41 | luyao | dansmith: Great, I'll give my question first. It's about libvirt driver, it seems that it's not recommended to access DB in libvirt driver, but in my patch, I need to get flavor id from db to populate the device manager, is that acceptable? I would like you help to give some comments. | |
| 14:12:43 | spatel | It clearly related to resize issue or bug | |
| 14:13:42 | sean-k-mooney | ya i have seen that before. | |
| 14:13:54 | sean-k-mooney | are you deploying on nfs or shared stoage | |
| 14:14:04 | dansmith | luyao: no, you can't access the database directly, but the flavor is on the instance so it should already be there and have what you need | |
| 14:15:15 | sean-k-mooney | spatel: this can happen when an instance was delete in the db when the compute agent is stopped and then was archived before it was started again | |
| 14:15:50 | sean-k-mooney | e.g. after an evacutation | |